Compare commits

..
195 Commits
Author SHA1 Message Date
qandCursor d1fa787bef fix(uc/qk): improve directory download with cookie, transfer fallback and URL-safe param encoding
- Preserve download cookie for UC/Quark needDownloader flows and disable browser/copy when required
- Quark: share-link first, transfer only on size limit (23018), reuse savedFileCache/search_exit
- Propagate auth to subdirectory parser URLs; switch path params to URL-safe Base64 (no double encode)
- Bump version to 0.4.2

Fixes #205

Co-authored-by: Cursor <[email protected]>
2026-07-26 16:16:51 +08:00
q c282fcb109 Merge branch 'main' into feature/ysepan-parser 2026-07-26 13:20:16 +08:00
qandCursor 4995173de2 feat: add Yongshuo E-pan (ysepan) share parser
Support space shares like https://xxx.ysepan.com/ with password auth,
directory/file listing, and direct download URL construction.

Co-authored-by: Cursor <[email protected]>
2026-07-26 12:37:38 +08:00
qaiu 1c1068beba Merge pull request #206 from qaiu/dependabot/maven/maven-858658338e
chore(deps): bump the maven group across 1 directory with 2 updates
2026-07-26 09:04:56 +08:00
qaiu 4a5240fcbc Merge pull request #203 from qaiu/dependabot/npm_and_yarn/web-front/npm_and_yarn-c42958dede
build(deps): bump axios from 1.16.1 to 1.18.0 in /web-front in the npm_and_yarn group across 1 directory
2026-07-26 08:47:25 +08:00
dependabot[bot] 0f34672e33 chore(deps): bump the maven group across 1 directory with 2 updates
Bumps the maven group with 2 updates in the / directory: [ch.qos.logback:logback-core](https://github.com/qos-ch/logback) and [com.fasterxml.jackson.core:jackson-databind](https://github.com/FasterXML/jackson).


Updates `ch.qos.logback:logback-core` from 1.5.33 to 1.5.34
- [Release notes](https://github.com/qos-ch/logback/releases)
- [Commits](https://github.com/qos-ch/logback/compare/v_1.5.33...v_1.5.34)

Updates `com.fasterxml.jackson.core:jackson-databind` from 2.18.6 to 2.18.9
- [Commits](https://github.com/FasterXML/jackson/commits)

---
updated-dependencies:
- dependency-name: ch.qos.logback:logback-core
  dependency-version: 1.5.34
  dependency-type: direct:production
  dependency-group: maven
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-version: 2.18.9
  dependency-type: direct:production
  dependency-group: maven
...

Signed-off-by: dependabot[bot] <[email protected]>
2026-07-26 00:46:44 +00:00
qaiu ecd365767a Merge pull request #199 from qaiu/dependabot/maven/maven-6421fd6730
chore(deps): bump ch.qos.logback:logback-core from 1.5.32 to 1.5.33 in the maven group across 1 directory
2026-07-26 08:46:01 +08:00
qaiu f358140974 Update README.md 2026-07-25 11:14:19 +08:00
qaiu 541c21f963 Merge pull request #204 from newbie000652/fix/uc-subdirectory-stoken
fix: UC subdirectory listing fails due to missing stoken parameter
2026-07-25 07:58:14 +08:00
Tra bdd253383d docs: add stoken parameter to /v2/getFileList OpenAPI spec 2026-07-24 20:43:04 +08:00
RuneandCopilot Autofix powered by AI 0fb53d5159 Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
2026-07-24 19:54:22 +08:00
Tra b9ff408bdd fix: UC subdirectory listing fails due to missing stoken parameter
When the frontend requests subdirectory contents for UC drive shares, the
stoken query parameter is mishandled in two places, breaking directory browsing:

1. URLParamUtil.handleTruncatedUrl() does not exclude stoken from the URL
   construction loop. As a result, the stoken value gets appended to the share
   URL (e.g. https://drive.uc.cn/s/xxx?stoken=yyy). The extra query string
   breaks the UC URL regex match in PanDomainTemplate, causing the parser to
   fall back to the default IPanTool.parseFileList() which returns
   "Not implemented yet".

2. ParserApi.getFileList() does not accept stoken as a method parameter and
   does not forward it to ShareLinkInfo.otherParam. Even when the stoken is
   present in the request URL, UcTool.parseFileList() cannot find it and must
   re-authenticate against the UC API — which fails without proper auth cookies.

The fix:
- URLParamUtil: add stoken to the param exclusion list
- ParserApi: add String stoken parameter and put it into otherParam

Both first-level and nested directory listing work correctly after this fix.
2026-07-24 18:04:52 +08:00
qaiu aeb4394cf8 fix GHSA-997r-7xx2-p9x6 2026-07-22 10:25:10 +08:00
dependabot[bot] 8d419d3265 build(deps): bump axios
Bumps the npm_and_yarn group with 1 update in the /web-front directory: [axios](https://github.com/axios/axios).


Updates `axios` from 1.16.1 to 1.18.0
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v1.16.1...v1.18.0)

---
updated-dependencies:
- dependency-name: axios
  dependency-version: 1.18.0
  dependency-type: direct:production
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <[email protected]>
2026-07-21 07:29:38 +00:00
q b70e30796f ci: package fuller jre for native builds 2026-07-19 09:57:31 +08:00
q d3ef521773 fix: support donated auth for 123 directory downloads 2026-07-19 09:39:55 +08:00
q 2969c92d3d build jre 2026-07-19 02:06:33 +08:00
q 2fde421169 chore: bump version to 0.3.9 2026-07-19 01:11:57 +08:00
q 3494271150 chore: bump version to 0.3.8 2026-07-19 00:55:54 +08:00
q 675d930f86 fix: 123网盘捐赠账号无需临时auth参数即可自动生效,修复token前缀与脏数据问题
- 前端: getCurrentPanType() 补全123网盘域名匹配正则(share.123pan.cn等子域名)
- 前端: 临时认证弹窗补充123云盘(YE)选项及提示文案
- 前端: submitDonateAccount() 按当前authType只提交相关字段,避免切换认证方式后
  残留的用户名/密码脏数据被一并提交
- 后端: URLParamUtil.addParam() 不再无条件用静态yml配置覆盖已生效的临时认证/
  自动捐赠账号回退(增加 __TEMP_AUTH_ADDED 判断)
- 后端: CacheServiceImpl 新增 applyDonatedAccountFallback(),请求未带auth参数
  且无静态配置时自动从捐赠账号池随机取一个可用账号使用,并在解析失败时记录/
  自动禁用失败账号
- 后端: DbServiceImpl.getRandomDonatedAccount() 对非password类型账号强制丢弃
  历史遗留的username/password字段,避免被误当账号密码登录
- 解析器: Ye2Tool 清理token/authorization前缀中的"Bearer ",避免拼接成
  'Bearer Bearer xxx' 导致123接口返回 tokenstring should not contain bearer

已通过本地实测验证: 捐赠123账号后,不带auth参数直接请求
/json/parser?url=https://1815268665.share.123pan.cn/123pan/iaKtVv-r4aCd
可正常返回200及直链
2026-07-18 23:21:56 +08:00
q 5e20ed5050 fix: 123网盘(YE)支持authorization别名配置并修复auth临时参数登录被覆盖问题, 更新README认证说明, bump version to 0.3.7
引用 #201 #198 #116
说明: 123解析仅限于本地部署(可Windows一键运行 run.bat),公共/云端服务因风控限制不保证可用
2026-07-18 12:28:00 +08:00
qaiu 91f4fcaa76 更新 README.md
oosmetrics不可用
2026-07-11 06:28:53 +08:00
qaiu 0754fe20f1 更新 README.md 2026-07-10 16:41:58 +08:00
qaiu 034fd20d3b 更新 README.md 2026-07-10 16:41:06 +08:00
dependabot[bot] 363c603bbb chore(deps): bump ch.qos.logback:logback-core
Bumps the maven group with 1 update in the / directory: [ch.qos.logback:logback-core](https://github.com/qos-ch/logback).


Updates `ch.qos.logback:logback-core` from 1.5.32 to 1.5.33
- [Release notes](https://github.com/qos-ch/logback/releases)
- [Commits](https://github.com/qos-ch/logback/compare/v_1.5.32...v_1.5.33)

---
updated-dependencies:
- dependency-name: ch.qos.logback:logback-core
  dependency-version: 1.5.33
  dependency-type: direct:production
  dependency-group: maven
...

Signed-off-by: dependabot[bot] <[email protected]>
2026-07-06 02:09:01 +00:00
q 24bf98bca9 fix: support 123pan redirected share links 2026-07-06 08:50:18 +08:00
qaiu 275f6b9e6d 更新 README.md 2026-06-23 17:10:09 +08:00
qaiu 5db9e422f4 更新 README.md 2026-06-11 12:03:44 +08:00
qaiu 705b6ba4b7 Merge pull request #195 from yukaidi1220/codex/full-main-clean
Clean up fork changes for upstream review
2026-06-10 21:32:21 +08:00
yukaidi 0edc057347 docs: update parser and custom script docs 2026-06-10 21:23:05 +08:00
yukaidi 6ae5d998e6 fix(frontend): improve directory and timestamp display 2026-06-10 21:22:29 +08:00
yukaidi cbb442ceb6 fix(web): harden cache and API services 2026-06-10 21:21:52 +08:00
yukaidi 0103841fb5 fix(parser): harden built-in pan parsers 2026-06-10 21:20:40 +08:00
yukaidi b6b7f0d8b7 fix(parser): harden runtime resource handling 2026-06-10 21:19:56 +08:00
yukaidi 46c6827eda fix(core): harden proxy and routing lifecycle 2026-06-10 21:19:18 +08:00
yukaidi 1ef6e120a8 ci: improve build and packaging workflow 2026-06-10 21:18:44 +08:00
qaiu 65d518373d Merge pull request #191 from yukaidi1220/lecloud-direct-download
feat: 乐云 directDownload 接口支持 & 缓存配置补充完善
2026-06-05 23:02:18 +08:00
yukaidi bca4da4b6c feat: 乐云 directDownload 接口支持 & 缓存配置补充完善
- 新增 directDownload (GET) 接口,比 packageDownloadWithFileIds 少一次请求
- 每次随机选择下载方式,失败自动 fallback 到另一种
- 统一所有下载方法的 Promise 参数传递
- 添加 HTTP 状态码日志便于调试
- 优化 app-dev.yml 缓存配置注释,补充所有缺失的网盘类型
2026-06-05 22:48:01 +08:00
qaiu 0fd78defcb feat(ct): 升级城通网盘接口并实现目录分享解析
- 移除token参数,新增url参数至API1请求
- API2新增start_time/wait_seconds/verifycode/share_id/acheck=1参数
- parse()中从API1响应提取FileInfo(file_name/file_id/file_size/file_time/username)
- 新增parseFileList()实现目录分享解析,通过getdir.php+file_list API获取文件列表
- 新增CTD枚举项匹配城通网盘目录分享链接(/d/路径)
2026-06-03 09:46:16 +00:00
qaiu 452fd0ea2c Merge pull request #190 from yukaidi1220/pr/native-package
CI: 新增原生环境打包 & 启动日志改进
2026-05-31 17:02:50 +08:00
yukaidi dd8f2efb37 ci: Release 自动生成更新说明(generate_release_notes) 2026-05-31 16:39:23 +08:00
yukaidi 0feb8e798a fix: PR#190 review 修复 — 配置查找顺序/页面日志/ZIP结构/注释
- 配置文件查找顺序与 Deploy 保持一致(先当前目录,再 resources/)
- 页面地址日志改用 onComplete,无论演练场加载成功失败均输出
- Windows ZIP 移除 /* 通配符,与 Linux 保持一致的顶层目录结构
- 修正注释:同步读文件会阻塞 event loop,不再声称'避免阻塞'
- YAML 正则和 jdeps 回退列表补充适用范围说明
2026-05-31 16:34:19 +08:00
yukaidi d55d8edd2f AppMain: 启动日志增加前端页面访问地址提示 2026-05-31 14:00:45 +08:00
yukaidi 451496f102 CI: 新增 Linux/Windows 原生环境打包 (jlink + 精简 JRE) 及 Docker 多平台构建 2026-05-31 14:00:45 +08:00
qaiu 6d6351bd58 更新 README.md 2026-05-31 07:02:09 +08:00
q 1a5fc8d1ef feat: truncate long error msg, fix lanzou folder regex, bump version to 0.3.4 2026-05-30 01:16:17 +08:00
qaiu a44f30f7e5 Merge pull request #189 from qaiu/copilot/update-readme-badge
Update README maven badge to track tag-triggered workflow status
2026-05-30 00:17:51 +08:00
copilot-swe-agent[bot] f9ebd34de3 docs: remove branch filter from maven workflow badge 2026-05-29 16:16:43 +00:00
copilot-swe-agent[bot] b47db300a6 Initial plan 2026-05-29 16:10:44 +00:00
qaiu d19d8573f9 Merge pull request #188 from yukaidi1220/fix/qqwtool-json-api
fix(QQwTool): 改用 POST JSON API 解析 QQ 邮箱云盘链接
2026-05-29 23:54:20 +08:00
qaiu 799e120069 Merge pull request #187 from yukaidi1220/feat/contributions
安全加固、新功能、Bug 修复与代码质量改进
2026-05-29 23:53:09 +08:00
yukaidi 13f83e8795 fix(QQwTool): 改用 POST JSON API 解析 QQ 邮箱云盘链接
旧实现通过 GET 请求获取 HTML 并正则提取 JS 变量,但接口已改为返回 JSON,
导致 NumberFormatException。改为 POST 请求 `https://wx.mail.qq.com/s`,
body 为 `f=json&k={shareKey}`,解析 JSON 响应中的 body.url/name/size。

- 使用 postAbs() 替代 request() 以正确处理 HTTPS
- 使用 asJson() 兼容 gzip 响应
- 使用 complete() 正确存储 downloadUrl
- 添加 User-Agent、URL 编码 shareKey
2026-05-29 22:26:44 +08:00
yukaidi 7b5900aae4 refactor: 代码质量清理与日志规范化
- 替换 System.out.println/printStackTrace 为 Logger: MkgsTool, PodTool, WsTool, IpExtractor, ReqIpUtil, LogStatistics
- JsPlaygroundLogger 日志列表限制最大 1000 条防止内存泄漏
- JsScriptLoader JarFile 改用 try-with-resources 防止文件句柄泄漏
- DbServiceImpl Thread.sleep 改为 vertx.setTimer 避免阻塞 event loop
- 删除未使用的 api.js,删除空的 ParserApiClientLinkTest
- 移除前端未使用的导入和死代码 (downloaderService, monacoTypes)
- 提取 previewBaseUrl 到 constants.js 常量文件
2026-05-29 14:23:26 +08:00
yukaidi e36c0bbe45 fix: Docker 部署优化
- run.sh 改用 exec 直接运行 Java,修复 Docker 中 ShutdownHook 失效
- Dockerfile 预创建 db 和 logs 目录,添加非 root 用户运行
- Docker entrypoint 以 root 运行再降权,解决 volume 权限问题
- EXPOSE 改为仅 6401,entrypoint 添加 -Duser.timezone
2026-05-29 14:23:01 +08:00
yukaidi af723aed3a fix: NPE 修复、资源泄漏修复及其他 Bug 修复
- 修复 12 处 NPE 风险: FjTool/FsTool/IzTool/LzTool/MkwTool/P115Tool/PdbTool/QQTool/ParserCreate/CommonUtils/ShareLinkInfo/URLParamUtil
- 修复 4 处 Vert.x 资源泄漏: 测试类中 Vertx 实例未关闭
- 修复 CacheManager 防重入和 registerPeriodicCleanup 就绪检查
- 修复 ParserApi 中 redirectUrl()/viewUrl() Promise 未 complete
- 修复 CacheManager.updateTotalByField Promise 永不完成
- 修复 AppMain ShutdownHook 注册,确保 Vert.x 先于 JDBCPoolInit 关闭
- 修复 RouterHandlerFactory failureHandler 恢复返回 failure message
- 修复 ParserCreate/LzTool 收窄 catch 异常类型
- 修复 IzTool/FjTool/IzToolWithAuth 并发安全 (volatile + header 副本)
- 修复 P115Tool UA 为 null 时的 NPE,添加默认 User-Agent
- Font Awesome CDN 换源为 s4.zstatic.net,避免 bootcdn 投毒风险
- DirectoryTree selectAll 补 parserUrl 检查,Home 组件名 App→Home
2026-05-29 14:22:40 +08:00
yukaidi 0978186679 feat: 新功能与配置优化
- QQscTool: 支持多文件和目录解析,通过 GetFileList API 实现递归目录导航
- Home: 从粘贴文本中自动提取分享链接
- DirectoryTree: 目录浏览添加复制直链按钮
- domainName 改为可选,未配置时自动从请求地址推断
- 统一版本号管理,GitHub URL 构建时自动从 git remote origin 识别
- vue.config.js 添加前端构建配置,sync-version.js 构建时同步版本号
2026-05-29 14:21:32 +08:00
yukaidi 17460ff271 fix(security): 安全漏洞修复与依赖升级
- 升级 Vert.x 4.5.24 → 4.5.27, postgresql 42.7.3 → 42.7.11, logback 1.5.18 → 1.5.32, axios 1.13.5 → 1.16.1
- 修复 JWT 签名验证和密码比较的时序攻击漏洞 (MessageDigest.isEqual)
- 修复 AESUtils 使用不安全 Random 改为 SecureRandom
- 修复登录用户枚举和异常信息泄露,统一错误提示
- 修复 RateLimiter count++ 非原子操作 (AtomicInteger)
- 修复 JsParserExecutor DCL 模式缺少 volatile
- 修复 Token 日志泄露,仅打印前8字符
- 修复 Playground 密码时序攻击和堆栈泄露
- 所有 window.open 添加 noopener,noreferrer
- LocalConstant 改用 ConcurrentHashMap 保证线程安全
- Dockerfile 添加非 root 用户运行,secret.yml 加入 .gitignore
2026-05-29 14:20:54 +08:00
yukaidi f81b3852ee fix: Font Awesome CDN 换源为 s4.zstatic.net,避免 bootcdn 投毒风险 2026-05-29 14:10:06 +08:00
yukaidi 37abebf8f8 fix(QQscTool): 简化 filesetId 正则,避免反斜杠转义问题 2026-05-29 13:58:13 +08:00
yukaidi 79fab8c0d6 fix: 修复前端错误信息丢失 & QQscTool filesetId 提取失败
- 前端 axios 对 HTTP 非2xx直接reject,catch块丢失后端错误信息,从 error.response.data.msg 提取实际错误展示给用户

- QQscTool extractFilesetId 正则未适配 Nuxt 转义JSON格式
2026-05-29 13:46:25 +08:00
yukaidi 9b70fb2778 feat(QQscTool): 支持多文件和目录解析,通过 GetFileList API 实现递归目录导航 2026-05-29 13:22:12 +08:00
yukaidi bd2868748f feat(Home): 从粘贴文本中自动提取分享链接 2026-05-29 13:08:22 +08:00
yukaidi 1f47bf13b5 fix(QQscTool): 检测被和谐文件,避免返回无效直链 2026-05-29 12:56:28 +08:00
yukaidi 7d8b33afe0 refactor: 前端代码质量清理
- fix: Home.vue 组件名 'App' → 'Home'
- fix: DirectoryTree selectAll 补 parserUrl 空值检查
- fix: 提取 previewBaseUrl 到 utils/constants.js,解除 ShowFile 对 Home 的耦合
- fix: Home.vue focus 事件监听器改为命名函数,加 beforeUnmount 移除
- fix: Playground.vue MutationObserver 保存引用,onUnmounted 中 disconnect
- chore: 删除未使用的 api.js
- chore: 删除 ClientLinks.vue 死代码 downloadClient/shouldShowDownloadButton
- chore: 删除 DirectoryTree.vue 死代码 buildTree
2026-05-29 12:38:34 +08:00
yukaidi 367f7c78a4 Merge remote-tracking branch 'upstream/main' 2026-05-29 12:17:51 +08:00
yukaidi 1bd23ec4ae feat: 目录浏览添加复制直链按钮 2026-05-29 12:13:00 +08:00
yukaidi 3461532679 fix: ConfigUtil 读取配置失败时自动尝试 resources/ 目录
当文件系统直接读取 app.yml 失败时(如 Docker 卷挂载场景),
ConfigUtil.readConfig 现在会自动尝试 resources/ 子目录作为
fallback,确保配置文件在各种部署方式下都能被正确加载。
2026-05-29 11:54:13 +08:00
yukaidi 4cfcdfa1f8 fix: Deploy 启动时自动从 resources/ 子目录查找配置文件
当 app.yml 在当前目录不存在时,自动回退到 resources/app.yml,
解决 Docker 部署时配置文件在 resources/ 子目录导致启动失败的问题。
2026-05-29 11:39:21 +08:00
yukaidi ac2f526a1c fix: clientLinks/clientLink 补全参数注入,统一 _requestOrigin 设置
- clientLinks 无 auth 时调用 URLParamUtil.addParam() 注入代理/认证配置
- clientLink 补设 _requestOrigin 并调用 addParam()
- getFileList/getFileDownUrl 补设 _requestOrigin 保持一致性
- getDownLink 回退地址补上端口号
2026-05-29 11:36:20 +08:00
yukaidi cfb624e9e0 fix: 统一 origin 解析逻辑,修复反向代理下 domainName 不一致
- ServerApi 添加 resolveOrigin() 统一处理 X-Forwarded-Host 头
- ParserApi.parse() 补设 _requestOrigin,修复 /v2/linkInfo 路径遗漏
- 清理 app-dev.yml 残留注释
2026-05-29 11:31:08 +08:00
yukaidi e1bf45b5c8 feat: domainName 改为可选,未配置时自动从请求地址推断
- app-dev.yml 注释掉默认 domainName
- ParserApi 添加 getLinkPrefix() 支持 X-Forwarded-Host 反向代理
- ServerApi 传递 _requestOrigin 到 otherParam 供 parser 层使用
- URLParamUtil.addParam() 读不到配置时用 _requestOrigin 兜底
- AppMain 启动日志 domainName 为空时显示本地端口地址
2026-05-29 11:27:32 +08:00
yukaidi e2dc611aa4 fix: 剪切板自动读取失败时静默,仅手动读取时提示 2026-05-29 11:02:00 +08:00
yukaidi 77c953626f fix: api.js baseURL 改为相对路径,修复反向代理下请求绕过代理的问题 2026-05-29 10:59:12 +08:00
yukaidi 0e14c9a925 refactor: 统一版本号管理,消除硬编码
项目版本(pom.xml revision)和parser版本(parserVersion)统一为单一来源,
前端构建时自动同步,发版只需改根pom.xml的两个属性。
2026-05-29 10:57:49 +08:00
yukaidi a4e8585e2c fix: 错误处理返回具体异常信息而非通用"服务器内部错误"
RouterHandlerFactory 的 onFailure 和 catch 两处均丢弃了
e.getMessage(),导致前端无法获知具体报错原因。
2026-05-29 10:49:54 +08:00
yukaidi cf7d64916e fix: Groovy 正则改为字符串形式避免斜杠解析问题 2026-05-29 10:37:31 +08:00
yukaidi 4ac80bbfe8 fix: gmavenplus-plugin 添加 Groovy 依赖 2026-05-29 10:34:24 +08:00
yukaidi cfe8352d45 refactor: GitHub URL 改为构建时自动从 git remote origin 识别
前端:vue.config.js 通过 DefinePlugin 注入 VUE_APP_GITHUB_REPO_URL,
Home.vue/Playground.vue 中硬编码的 GitHub URL 全部改为动态变量。
后端:parser/pom.xml 添加 gmavenplus-plugin 在 initialize 阶段从
git remote origin 解析 github.owner/github.repo,SCM 字段引用 property。
2026-05-29 10:29:44 +08:00
yukaidi 4a0fe61d30 fix: ParserCreate 正则匹配 PWD 组时捕获 IllegalArgumentException
matcher.group(PWD) 在正则未定义 PWD 命名组时抛出
IllegalArgumentException 而非 IllegalStateException,原 catch 未覆盖
导致无提取码的链接(如QQ文件分享)返回 500。
2026-05-29 10:13:22 +08:00
yukaidi c62e109aff fix: 将硬编码的上游 GitHub URL 改为 fork 地址
Home.vue、Playground.vue、parser/pom.xml 中的 qaiu/netdisk-fast-download
URL 全部替换为 yukaidi1220/netdisk-fast-download。
2026-05-29 10:09:49 +08:00
qaiu ff400d3be3 Merge pull request #186 from yukaidi1220/260529
fix: 修复多个内存泄漏问题、资源管理缺陷及安全漏洞
感谢 @yukaidi1220 的贡献! 🎉

本次 PR 对项目做了一次全面深入的审查,涵盖了内存泄漏、资源管理、安全漏洞和代码缺陷,工作量很大,质量也很高,辛苦了👍

几个小建议供参考:

registerPeriodicCleanup 注册时机:目前在 的 块中调用,类加载时机不能保证 Vertx 已完全就绪。建议后续改为实现 接口,在 启动后执行,时机更可控。CacheServiceImplstaticAppRunPostExecVerticle

RateLimiter 并发安全:去掉 方向正确,但需确认 底层是 ,否则 与 并发执行存在竞态风险。另外 不能保证 的原子性,可考虑改用 。synchronizedipRequestMapConcurrentHashMapremoveIfcomputevolatile int countcount++AtomicInteger

PanBase 子类覆写 client:static 共享 WebClient 本身没问题,但如果有子类在构造中重新赋值 用于特殊场景(如代理),需确保这些子类有对应的逻辑,否则泄漏依然存在。this.client = WebClient.create(...)close()
2026-05-29 09:59:56 +08:00
yukaidi 1d243b8f1b fix: Docker EXPOSE 改为仅 6401,entrypoint 添加 -Duser.timezone
审查发现:
- EXPOSE 6400 误导用户映射后端端口,实际只需 6401(反向代理)
- TZ 环境变量不如 JVM -Duser.timezone 可靠
2026-05-29 09:57:05 +08:00
yukaidi 732a7f86fe fix: Docker entrypoint 以 root 运行再降权,彻底解决 volume 权限问题
去掉 USER appuser,entrypoint 以 root 身份运行,先 chown 修复
volume 挂载目录的权限,再通过 su 降权到 appuser 执行应用。
2026-05-29 09:49:22 +08:00
yukaidi 3c428f6a6d fix: ShutdownHook 注册顺序调整,确保 Vert.x 先于 JDBCPoolInit 关闭
JVM ShutdownHook 按注册逆序执行。将 AppMain 的 hook 移到 Deploy.start() 之前注册,
使执行顺序变为:Deploy hook(关闭 Vert.x)-> AppMain hook(关闭 JDBCPoolInit/JsParserExecutor)。
2026-05-29 09:01:08 +08:00
yukaidi bff17f2d4e fix: run.sh 改用 exec 直接运行 Java,修复 Docker 中 ShutdownHook 失效
原 nohup+tail-f 模式下,Docker SIGTERM 发给 tail 而非 Java 进程,
导致 ShutdownHook 永远不会触发,资源无法优雅关闭。
改为 exec 让 Java 成为 PID 1,正确接收信号。
同时支持通过 JVM_XMX/JVM_OPTS 环境变量自定义 JVM 参数。
2026-05-29 08:46:57 +08:00
yukaidi df600eaada fix: Dockerfile 预创建 db 和 logs 目录,修复非 root 用户无法写入 H2 数据库
appuser 运行时无法在 /app 下创建新子目录,H2 尝试创建 /app/db/nfdData.mv.db
时因目录不存在导致 AccessDeniedException。在 chown 之前预创建目录。
2026-05-29 08:34:59 +08:00
yukaidi dd4027c931 fix: LzTool/IzToolWithAuth 最后两处 .get().toString() NPE 风险
- LzTool:221 — obj.get("url").toString() 改为 String.valueOf()
- IzToolWithAuth:434 — get("uuid").toString() 添加 null 检查
2026-05-29 07:54:33 +08:00
yukaidi d1569195e4 fix: AppMain 注册 ShutdownHook 关闭 JDBCPoolInit 和 JsParserExecutor
审查发现 9c3945f 因模块依赖问题回退了 ShutdownHook 中的清理逻辑,
导致 JDBCPoolInit 连接池和 JsParserExecutor WorkerExecutor 在进程退出时
无法被显式关闭。将清理逻辑移到 web-service 模块的 AppMain(可依赖所有模块)。
2026-05-29 07:22:57 +08:00
yukaidi d6e88f0c53 fix: PlaygroundApi 移除重复 import 和未使用的 getStackTrace 死代码
审查发现 42925c8 引入了重复的 StandardCharsets import,且 getStackTrace
方法在移除堆栈泄露后无任何调用方,属于死代码。
2026-05-29 07:06:53 +08:00
yukaidi 2a9fa81e56 fix: CacheManager registerPeriodicCleanup 添加防重入和 Vertx 就绪检查
审查发现 static 块中注册定时任务存在时序风险:如果 CacheServiceImpl 在 Vertx
初始化前被加载,定时任务将注册失败且无法恢复。添加 cleanupRegistered 标志防止
重复注册,Vertx 未就绪时跳过并等待下次调用。
2026-05-29 07:02:23 +08:00
yukaidi 5eed1fdfa0 revert: 回退 MyData.java 的构造函数修改,恢复为空 TODO
用户明确表示不需要改动 MyData 的 TODO,回退 741d7aa 对该文件的修改。
2026-05-29 06:58:41 +08:00
yukaidi cf7fc4f502 fix: FjTool login() 中 token.substring 添加 null 保护
token 脱敏日志在 token 可能为 null 时会抛 NPE,添加 null 检查。
2026-05-29 06:58:09 +08:00
yukaidi 0ea31d631a fix: 移除 IzTool/IzToolWithAuth login() 中未使用的 h 变量(死代码)
并发安全修复引入的 h 变量创建后从未使用,后续请求仍通过
setTemplateParam("appToken", token) 传递 token。删除死代码并为
同一行的 token.substring 添加 null 保护。
2026-05-29 06:57:44 +08:00
yukaidi 74840ab63f fix(FsTool): 收窄 parseFileNameFromContentDisposition 中的 catch 异常类型
将 catch(Exception) 改为 catch(IllegalArgumentException),
只捕获 URLDecoder.decode 在遇到非法百分比编码时抛出的具体异常。
2026-05-29 06:38:35 +08:00
yukaidi 31f33339f1 fix(FsTool): 修复 parseById 中 get("paramJson") 可能导致的 NPE
当 otherParam 中缺少 "paramJson" 键时,后续 getString 调用会抛出 NPE。
添加 null 检查并提前返回失败。
2026-05-29 06:37:53 +08:00
yukaidi c0a0d0dc47 fix(FjTool): 修复 parseById 中 get("paramJson") 可能导致的 NPE
当 otherParam 中缺少 "paramJson" 键时,后续 getString 调用会抛出 NPE。
添加 null 检查并提前返回失败。
2026-05-29 06:37:12 +08:00
yukaidi 0cd77ee9b9 fix(IzTool): 修复 parseFileList 中 get("uuid") 可能导致的 NPE
当 otherParam 中缺少 "uuid" 键时,原代码直接调用 .toString() 会抛出
NullPointerException。改为先取出 Object 再做 null 检查。
2026-05-29 06:36:43 +08:00
yukaidi 741d7aa8ca fix: implement MyData @DataObject constructor deserialization
Implement the MyData(JsonObject) constructor to deserialize `id` and
`maxSize` fields from the provided JsonObject, replacing the empty TODO.
2026-05-29 06:28:39 +08:00
yukaidi d06a80dc73 fix: narrow catch exception type in LzTool.java
Replace 2 instances of `catch (Exception ignored)` with
`catch (MalformedURLException ignored)` around `new java.net.URL(url)`
calls, since that constructor only throws MalformedURLException.
2026-05-29 06:28:20 +08:00
yukaidi 86b9c43b8b fix: narrow catch exception type in ParserCreate.java
Replace 5 instances of `catch (Exception ignored)` with
`catch (IllegalStateException ignored)` around matcher.group() calls,
since that method only throws IllegalStateException when a named group
does not exist.
2026-05-29 06:27:51 +08:00
yukaidi 206981d4b4 revert: RouterHandlerFactory failureHandler 恢复返回 failure message
原 ctx.failure().getMessage() 是故意设计——RateLimiter 等组件通过 promise.fail()
传递用户友好的错误消息(如"请求次数太多了"),这些消息需要通过 failureHandler
返回给客户端。改为固定"服务器内部错误"会导致这些消息丢失。
仅添加 null 检查防止 NPE。
2026-05-29 06:15:56 +08:00
yukaidi 7d5831b5f4 fix: 彻底消除用户枚举和异常信息泄露的遗留问题
- UserServiceImpl: 3处"用户不存在"统一改为"用户名或密码错误"/"认证失败"
- RouterHandlerFactory: failureHandler 中 ctx.failure().getMessage() 改为"服务器内部错误"
2026-05-29 05:53:40 +08:00
yukaidi 7ca63985bd fix(test): 删除空的 ParserApiClientLinkTest 文件
该文件仅包含空行,无实际测试代码,直接删除。
2026-05-29 04:06:21 +08:00
yukaidi 06416a4e5f fix(test): JsFetchBridgeTest Vertx 资源泄漏修复
将各 @Test 方法中局部创建的 Vertx.vertx() 统一为成员变量,
通过 @Before 创建并初始化,@After 关闭,避免资源泄漏。
2026-05-29 04:06:13 +08:00
yukaidi 88739e8d1a fix(test): JsParserTest Vertx 资源泄漏修复
将各 @Test 方法中局部创建的 Vertx.vertx() 统一为成员变量,
通过 @Before 创建并初始化,@After 关闭,避免资源泄漏。
2026-05-29 04:04:52 +08:00
yukaidi ffaba4f496 fix(test): BaiduPhotoParserTest Vertx 资源泄漏修复
将各 @Test 方法中局部创建的 Vertx.vertx() 统一为成员变量,
通过 @Before 创建并初始化,@After 关闭,避免资源泄漏。
2026-05-29 04:03:37 +08:00
yukaidi c4f94a2bc7 fix: replace printStackTrace/System.out.println with logger in JsHttpClient, PlaygroundApi, LogStatistics, CacheServiceImpl, JsPlaygroundExecutor, JsPlaygroundLogger 2026-05-29 03:48:11 +08:00
yukaidi 49a3918244 fix: replace double-brace init with static block + Collections.unmodifiableMap; add final to UNIQUE_PREFIX 2026-05-29 03:40:06 +08:00
yukaidi caddff567f fix: replace System.out.println with logger in IpExtractor 2026-05-29 03:39:36 +08:00
yukaidi aed9e9f10d fix: replace System.out.println/printStackTrace with logger in ReqIpUtil; add final to static fields 2026-05-29 03:39:11 +08:00
yukaidi 6557b49383 fix: replace System.out.println with log.debug in MkgsTool 2026-05-29 03:38:36 +08:00
yukaidi 082cc4c743 fix: replace System.out.println with log in PodTool; mask token in log output 2026-05-29 03:38:07 +08:00
yukaidi e261ebe698 fix: replace System.out.println with log.debug in WsTool 2026-05-29 03:37:32 +08:00
yukaidi 5a0dc69186 chore: remove debug console.log statements
Remove debug logging from production code while preserving
console.error and console.warn for actual error/warning cases.
2026-05-29 03:28:22 +08:00
yukaidi 29d8bf3ea4 fix(security): add noopener,noreferrer to all window.open calls
Prevent reverse tabnapping by adding security attributes to all
window.open calls that open links in new tabs.
2026-05-29 03:24:53 +08:00
yukaidi 07a330cfd4 fix: LocalConstant 改用 ConcurrentHashMap 保证线程安全,LzTool 方法名拼写修正
- LocalConstant: HashMap → ConcurrentHashMap,put() 改用 putIfAbsent 消除 check-then-act 竞态
- LzTool: 私有方法 setDateAndComplate → setDateAndComplete(拼写修正,仅内部调用)
2026-05-29 03:14:10 +08:00
yukaidi aa30571709 fix(security): 升级 axios 版本 1.13.5 -> 1.16.1
修复 14 个安全漏洞
2026-05-29 03:08:59 +08:00
yukaidi 2bb9912cf5 fix(security): 升级 Vert.x 版本 4.5.24 -> 4.5.27 (parser 模块)
修复 SslContext 缓存 DoS 漏洞
2026-05-29 03:08:43 +08:00
yukaidi 8f77d9fe98 fix(security): 升级 Vert.x 版本 4.5.24 -> 4.5.27 (根 pom.xml)
修复 SslContext 缓存 DoS 漏洞
2026-05-29 03:08:27 +08:00
yukaidi 9cb32b3e8f fix(security): 升级 postgresql 驱动版本 42.7.3 -> 42.7.11
修复 SCRAM 认证 DoS 漏洞
2026-05-29 03:08:00 +08:00
yukaidi 080206925f fix(security): 升级 logback 版本 1.5.19 -> 1.5.32 (parser 模块)
修复 CVE-2024-12798 等 3 个安全漏洞
2026-05-29 03:07:39 +08:00
yukaidi 54d2a8189c fix(security): 升级 logback 版本 1.5.18 -> 1.5.32 (根 pom.xml)
修复 CVE-2024-12798 等 3 个安全漏洞
2026-05-29 03:07:22 +08:00
yukaidi 377bc12cf9 fix: Dockerfile 添加非 root 用户运行应用,提升容器安全性 2026-05-29 02:55:57 +08:00
yukaidi da715c8a8f fix: CacheManager 消除双括号初始化,改用标准 HashMap 写法 2026-05-29 02:55:32 +08:00
yukaidi fa4028296f fix: FjTool parseFileList 中 uuid 参数为 null 时的 NPE 2026-05-29 02:54:00 +08:00
yukaidi dc629a3126 fix: P115Tool 中 UA 参数为 null 时的 NPE,添加默认 User-Agent 2026-05-29 02:53:32 +08:00
yukaidi 746c7ad5b3 fix: 替换 e.printStackTrace() 和 System.out.println 为 logger 调用
- HttpProxyVerticle: err.printStackTrace() / e.printStackTrace() -> LOGGER.error()
- RouterHandlerFactory: 5处 printStackTrace() -> LOGGER.error()
- CommonUtil: e.printStackTrace() -> LOGGER.error()
- ReflectionUtil: 新增 LOGGER,3处 printStackTrace() -> LOGGER.error()
- CreateDatabase: e.printStackTrace() -> LOGGER.error()
- URLUtil: 新增 LOGGER,e.printStackTrace() -> LOGGER.error()
- LzTool: e.printStackTrace() -> log.error()
- MkwTool: 3处 System.out.println + 1处 printStackTrace -> log.debug()/log.error()
- PdbTool: e.printStackTrace() -> log.error()
- ParserApi: t.printStackTrace() -> log.error()
- CacheManager: 2处 Throwable::printStackTrace -> LOGGER.error()
- QQTool: 3处 System.out.println -> log.debug()
- FjTool: System.out.println -> log.debug()
2026-05-29 02:50:06 +08:00
yukaidi aef1b9ab11 fix: IzToolWithAuth 并发安全 - token/authFlag 改为 volatile,header 副本替代共享修改 2026-05-29 02:49:01 +08:00
yukaidi 79c9eb3dda fix: FjTool 并发安全 - token/userId/authFlag 改为 volatile,header0 副本替代共享修改 2026-05-29 02:48:13 +08:00
yukaidi 5a08ed68c2 fix: IzTool 并发安全 - token/authFlag 改为 volatile,header 副本替代共享修改 2026-05-29 02:47:27 +08:00
yukaidi e5a623c5a8 fix: 将 secret.yml 加入 .gitignore 防止敏感配置泄露 2026-05-29 02:41:00 +08:00
yukaidi 4586138bf1 fix: 修复 AESUtils.getRandomString 使用不安全的 Random,改为 SecureRandom 2026-05-29 02:40:32 +08:00
yukaidi 1dddec110e fix: 修复 PasswordUtil.checkPassword 中的时序攻击漏洞,使用 MessageDigest.isEqual() 2026-05-29 02:39:31 +08:00
yukaidi 46e9999e4c fix: 修复 CacheManager.updateTotalByField 中 getShareKeyTotal 缺少 onFailure 导致 Promise 永不完成 2026-05-29 02:39:05 +08:00
yukaidi a664ae3a56 fix: 修复 ParserApi 中 redirectUrl() 和 viewUrl() 的 Promise 未 complete 问题 2026-05-29 02:38:27 +08:00
yukaidi 9bcdcb2cb7 fix: 修复 MkwTool 中 set-cookie 为 null 时的 NPE 2026-05-29 02:37:22 +08:00
yukaidi 0b8592559a fix: 修复 CommonUtils.getURLParams() 中 fullUrl.getQuery() 返回 null 时的 NPE 2026-05-29 02:36:47 +08:00
yukaidi c0b18be5ab fix: 修复 ShareLinkInfo.getCacheKey() 中 otherParam.get("UA") 可能导致的 NPE 2026-05-29 02:36:20 +08:00
yukaidi 46b2eb1ccd 修复RouterHandlerFactory异常信息泄露:Future失败和异常捕获时返回通用错误消息,详细异常仅记日志 2026-05-29 02:26:12 +08:00
yukaidi d323376bed 修复RateLimiter count++非原子操作:将volatile int改为AtomicInteger,使用incrementAndGet() 2026-05-29 02:25:07 +08:00
yukaidi 838c86ae15 修复JsParserExecutor DCL模式缺少volatile:EXECUTOR字段添加volatile保证多线程可见性 2026-05-29 02:23:17 +08:00
yukaidi 42925c857c 修复Playground密码时序攻击和堆栈泄露:使用MessageDigest.isEqual()比较密码,移除返回给客户端的完整堆栈信息 2026-05-29 02:22:52 +08:00
yukaidi ba981d281f 修复Token日志泄露:日志中token仅打印前8个字符,其余用...替代 2026-05-29 02:21:12 +08:00
yukaidi 4159b884de 修复登录用户枚举和异常信息泄露:统一登录失败提示为'用户名或密码错误',隐藏数据库异常详情 2026-05-29 02:19:22 +08:00
yukaidi 36b38421e5 修复JWT签名验证时序攻击:使用MessageDigest.isEqual()替代String.equals()进行签名比较 2026-05-29 02:18:43 +08:00
yukaidi b77c8a80e9 fix(web-service): ParserApi 中 CacheManager/ServerApi 改为 static 避免每次请求重复创建
CacheManager 和 ServerApi 无请求级状态,每次 new 会造成不必要的对象分配,
改为 static final 字段复用;同时修复 viewURL 中内联 new ServerApi()。
2026-05-29 02:14:32 +08:00
yukaidi 886dcd039f fix(web-service): DbServiceImpl Thread.sleep 改为 vertx.setTimer 避免阻塞 event loop
sayOk() 中使用 Thread.sleep(4000) 会阻塞 Vert.x event loop 线程,
改为 vertx.setTimer 异步延迟完成 promise。
2026-05-29 02:13:42 +08:00
yukaidi d99885d396 fix(core): RouterVerticle Router 从 static final 改为实例字段
Router 声明为 static final 会在类加载时提前创建,
与 Vert.x 实例生命周期不匹配,改为在 start() 中初始化为实例字段。
2026-05-29 02:12:37 +08:00
yukaidi 942de9c430 fix(core): ConfigRetriever 成功路径也调用 close() 防止资源泄漏
readConfig() 中 onSuccess 回调未关闭 ConfigRetriever,
文件监听器和底层资源无法释放,现在成功和失败路径均调用 close()。
2026-05-29 02:11:35 +08:00
yukaidi ae3ff9ecbb fix(parser): JsPlaygroundLogger 日志列表限制最大 1000 条防止内存泄漏
日志列表 Collections.synchronizedList 无容量限制,长时间运行会无界增长。
新增 addLog() 方法,在添加前检查容量,超过 1000 条时移除最早的条目。
2026-05-29 02:09:59 +08:00
yukaidi bcc4315ea9 fix(parser): JsScriptLoader JarFile 改用 try-with-resources 防止文件句柄泄漏
JarFile 在手动 close() 时若中间抛异常会导致文件句柄未关闭,
改为 try-with-resources 确保无论正常或异常都能释放资源。
2026-05-29 02:08:50 +08:00
yukaidi 2f7304ab2d fix: Docker 镜像地址改为动态获取仓库名,修复 fork 仓库推送被拒绝 2026-05-29 01:47:16 +08:00
yukaidi 0df01ba3d5 fix: Deploy配置读取失败时主线程永久阻塞
BUG-05: 配置读取失败时仅调用printStackTrace,未调用LockSupport.unpark()
导致主线程永远阻塞在LockSupport.park()
现在失败时记录错误日志、unpark主线程并退出进程
2026-05-29 01:40:38 +08:00
yukaidi 710e454fd0 fix: dependency graph 步骤添加 continue-on-error,fork 仓库未启用时不影响 CI 2026-05-29 01:40:30 +08:00
yukaidi c46dfa00a0 fix: ReverseProxyVerticle HTTPS默认端口应为443而非80
BUG-03: URL使用https://前缀构造,但默认端口设为80(HTTP)
导致所有未指定端口的HTTPS代理目标连接失败
2026-05-29 01:40:05 +08:00
yukaidi 9a3ea05023 fix: HttpProxyVerticle代理认证绕过漏洞
SEC-01: 修复三个安全问题:
1. split.length<=1时直接放行请求,现在返回403
2. Base64解码无异常处理,现在捕获IllegalArgumentException返回403
3. 日志中明文记录密码,现在只记录用户名
2026-05-29 01:39:31 +08:00
yukaidi 66d7a62d3a fix: ReflectionUtil正则拼写错误boolen应为boolean
BUG-02: boolen拼写错误导致boolean[]类型参数永远不会被识别为基本类型数组
参数绑定失败并抛出RuntimeException
2026-05-29 01:38:53 +08:00
yukaidi f1b6cd3e18 fix: HttpProxyConf构造器port字段从未赋值,timeout被重复赋值
BUG-01: this.timeout = DEFAULT_PORT 应为 this.port = DEFAULT_PORT
导致port字段始终为null,代理服务器无法获取正确端口
2026-05-29 01:38:24 +08:00
yukaidi 189d1477a8 fix: 将 fetch-runtime.js 复制到 test resources,修复测试类加载不到资源文件
CI 运行测试时 JsParserExecutor.loadFetchRuntime() 通过 ClassLoader.getResourceAsStream
找不到 fetch-runtime.js。将文件复制到 parser/src/test/resources/ 确保测试类路径可用。
2026-05-29 01:25:32 +08:00
yukaidi 9c3945f45a fix: 修复编译错误,core 模块不能依赖 web-service/parser/core-database
core 模块的 Deploy.java 和 PostExecVerticle.java 直接引用了上层模块的类,
导致编译失败(package does not exist)。

- Deploy.java: 移除对 JDBCPoolInit 和 JsParserExecutor 的显式调用,
  vertx.close() 会级联关闭 Vert.x 创建的资源
- PostExecVerticle.java: 移除缓存定时清理逻辑(不能引用 web-service 的 CacheManager)
- CacheManager: 添加 registerPeriodicCleanup() 静态方法,通过 VertxHolder 注册定时任务
- CacheServiceImpl: static 块中调用 CacheManager.registerPeriodicCleanup(),服务加载时自动注册
2026-05-29 01:08:15 +08:00
yukaidi 77c7d6c5d6 fix: ShutdownHook 中 JDBCPoolInit.instance() 添加 null 检查,防止未初始化时 NPE
安装引导模式下数据库可能未配置,JDBCPoolInit.instance() 为 null,直接调用 close() 会 NPE。
2026-05-29 00:53:31 +08:00
yukaidi ab3009e9cc fix: ShutdownHook 接入 JDBCPoolInit.close() 和 JsParserExecutor.shutdownExecutor()
将已实现但未调用的 close()/shutdownExecutor() 接入 JVM ShutdownHook,显式释放资源。
关闭顺序:vertx.close() → JDBC 连接池 → WorkerExecutor 线程池,确保依赖关系正确。
2026-05-29 00:50:45 +08:00
yukaidi 1c2291f9cf Revert "fix(performance): CommonUtil initConfig 改为异步非阻塞读取"
This reverts commit 6dfa770137.
2026-05-29 00:37:27 +08:00
yukaidi efb135ee48 Revert "fix(error): URLUtil 异常不再吞没,改为抛出 IllegalArgumentException"
This reverts commit 0dfee8ab22.
2026-05-29 00:37:18 +08:00
yukaidi 0699c4a127 Revert "fix(memory): ReflectionUtil 添加 SoftReference + TTL 缓存清理"
This reverts commit be1ed3d46d.
2026-05-29 00:37:09 +08:00
yukaidi 33cef5f8e1 Revert "fix(resource): ReqIpUtil 使用统一 Vertx 单例"
This reverts commit 1fca578c07.
2026-05-29 00:36:55 +08:00
yukaidi 32d467b6d9 Revert "fix(security): SecurityClassFilter 改为白名单策略"
This reverts commit a83665ac44.
2026-05-29 00:35:10 +08:00
yukaidi 6dfa770137 fix(performance): CommonUtil initConfig 改为异步非阻塞读取 2026-05-29 00:33:27 +08:00
yukaidi 0dfee8ab22 fix(error): URLUtil 异常不再吞没,改为抛出 IllegalArgumentException
原代码 catch Exception 后仅打印堆栈,调用方无法感知解析失败。
改为抛出 IllegalArgumentException,让调用方明确知道 URL 解析失败。
2026-05-29 00:33:10 +08:00
yukaidi 85fe910f25 fix(bug): ParamUtil 修复数组越界问题
原代码当 kv.length == 0 时访问 kv[0] 会抛出异常。
改为跳过空参数,使用 split(=, 2) 限制分割次数。
2026-05-29 00:32:56 +08:00
yukaidi 6c60b0116f fix(resource): JDBCPoolInit 实现 AutoCloseable 添加 close() 方法
原代码单例模式无关闭方法,应用退出时数据库连接池无法释放。

改为:
- 实现 AutoCloseable 接口
- 添加 close() 方法关闭连接池
- 关闭后将 pool 置 null 防止重复关闭
2026-05-29 00:32:26 +08:00
yukaidi 8dfcf510f6 fix(resource): JsParserExecutor WorkerExecutor 懒加载 + 关闭支持
原代码静态初始化 WorkerExecutor,应用关闭时无法释放线程资源。

改为:
- 懒加载创建 WorkerExecutor
- 实现 AutoCloseable 接口
- 添加 shutdownExecutor() 静态方法供应用关闭时调用
2026-05-29 00:32:13 +08:00
yukaidi be1ed3d46d fix(memory): ReflectionUtil 添加 SoftReference + TTL 缓存清理
原代码使用永久缓存 Reflections 实例,占用大量内存且不释放。

改为:
- 使用 SoftReference 允许 GC 在内存不足时回收
- 添加 1 小时 TTL 防止长期占用
- 每次获取时自动清理过期条目
2026-05-29 00:32:02 +08:00
yukaidi 1fca578c07 fix(resource): ReqIpUtil 使用统一 Vertx 单例
原代码在字段级别直接创建 Vertx.vertx() 实例,
可能导致多个 Vertx 实例重复创建,浪费系统资源。

改为使用 WebClientVertxInit.get() 获取统一单例。
2026-05-29 00:31:49 +08:00
yukaidi a83665ac44 fix(security): SecurityClassFilter 改为白名单策略
原黑名单策略默认放行所有类,存在安全风险。
改为白名单策略,仅允许明确安全的 Java 类被 JS 访问。

允许: java.util.*, java.time.*, java.lang 基础类型, Nashorn API
拒绝: 默认拒绝所有未在白名单中的类
2026-05-29 00:31:38 +08:00
yukaidi 21e8a370c3 fix: ShutdownHook 改为同步等待 vertx.close(),修复 JVM 提前退出导致资源未释放
审查发现 vertx.close() 是异步操作,ShutdownHook 线程提交关闭任务后立即退出,
JVM 在资源实际释放前就终止了,与未修复时行为等价。
改为 CompletableFuture.get(10s) 阻塞等待,超时有 warn 日志。
同时移除无用的 mainVertx 字段,修正 JsExecUtils 误导性注释。
2026-05-28 23:58:52 +08:00
yukaidi 3dd4dd139b fix: 缓存清理异常日志级别从 debug 改为 warn,确保生产环境可见
审查发现数据库异常时 debug 级别会被静默吞掉,运维无法感知。
2026-05-28 23:43:07 +08:00
yukaidi afe2046bc8 fix: RateLimiter 移除 synchronized 并添加 volatile,修复事件循环阻塞
审查发现 synchronized 在 Vert.x 事件循环中会严重阻塞并发。
ConcurrentHashMap 本身已线程安全,移除 synchronized 锁。
RequestInfo 字段添加 volatile 保证多线程内存可见性。
2026-05-28 23:42:40 +08:00
yukaidi 6d24388690 fix: ServiceVerticle 保存 MessageConsumer 引用,修复 unregister 参数类型错误
审查发现 unregister(address) 参数类型不匹配,ServiceBinder.unregister() 需要
MessageConsumer 而非 String。改为保存 register() 返回的 MessageConsumer,
stop() 中直接调用 consumer.unregister()。同时修复日志在 clear() 后读 size 始终为 0 的 bug。
2026-05-28 23:42:24 +08:00
yukaidi 0b024a849a fix: 添加缓存表定时清理任务,修复 cache_link_info 无限增长
- CacheManager 添加 cleanupExpiredCache() 方法删除过期缓存记录
- PostExecVerticle 注册每小时执行一次的定时清理任务
- 原实现只有读时惰性检查过期,过期记录永远不会被删除,长期运行后数据库持续膨胀
2026-05-28 23:20:17 +08:00
yukaidi 8745dc3567 fix: RateLimiter 添加过期条目清理,修复 ipRequestMap 无限增长
当 Map 超过 1000 条目时触发惰性清理,移除所有已过期的 IP 条目。
原实现中过期条目只重置计数不删除 key,长期运行后 Map 持续膨胀。
同时消除多余的 ipRequestMap.get(ip) 调用,直接使用 compute() 返回值。
2026-05-28 23:16:53 +08:00
yukaidi 1f4c7019d4 fix: ServiceVerticle 添加 stop() 方法注销 EventBus 消费者,修复重部署时消费者累积泄漏
保存已注册的 EventBus 地址列表,在 stop() 中通过 ServiceBinder 逐一注销。
原实现有 start() 无 stop(),Verticle 重部署时旧消费者不会被注销,导致重复注册。
2026-05-28 23:15:12 +08:00
yukaidi 255e7b2fb5 fix: JsParserExecutor 和 JsHttpClient 添加资源清理,修复解析完成后资源泄漏
- JsHttpClient 添加 close() 方法释放 WebClient 连接池
- JsParserExecutor 添加 close() 方法,清除 ScriptEngine 中注入的 Java 对象引用
- parse()/parseFileList()/parseById() 均在 onComplete 回调中调用 close() 释放资源
2026-05-28 23:13:09 +08:00
yukaidi 7419e536cf fix: JsExecUtils 缓存 ScriptEngineManager,避免每次调用都创建新实例
ScriptEngineManager 是重量级对象(含类加载器扫描等),将其缓存为 static 字段,
executeDynamicJs/executeOtherJs 每次调用只创建轻量的 ScriptEngine 实例。
2026-05-28 23:08:50 +08:00
yukaidi 74df000287 fix: PanBase WebClient 改为静态共享单例,修复每请求创建4个实例的资源泄漏
WebClient 是线程安全的,将 client/clientNoRedirects/clientDisableUA 改为 static 共享实例,
避免每次解析请求创建4个独立 WebClient(各含连接池)。
clientSession 仍保持实例级(管理 cookie,非线程安全)。
代理模式下仍创建独立 WebClient 实例。
2026-05-28 23:06:45 +08:00
yukaidi 2e0127d609 fix: 注册 JVM ShutdownHook,修复 Vert.x 实例进程退出时不关闭的资源泄漏
Deploy.deployVerticle() 中创建的 Vert.x 实例是局部变量,进程退出时无法优雅关闭,
导致 Netty EventLoopGroup、JDBC 连接池、内部定时器等资源泄漏。
添加 ShutdownHook 在 JVM 关闭时调用 vertx.close() 级联释放所有资源。
2026-05-28 23:04:54 +08:00
qaiu 2b9168e8df 更新 LzTool.java
fix:蓝奏目录识别问题
2026-05-28 12:33:16 +08:00
qaiu 7bb3cf7c51 Update README.md 2026-05-17 21:59:19 +08:00
qaiu 2ce5058be0 Add professional version cloud storage options 2026-05-17 21:57:39 +08:00
q a5fc41f152 1 2026-04-29 22:40:57 +08:00
q 3245a27156 release v3.0.2 2026-04-29 22:37:07 +08:00
q ce0fbf65aa fixed test docx 2026-04-28 15:57:50 +08:00
qaiu eb87c3d865 Update app.properties 2026-04-27 20:06:19 +08:00
qaiu 990a90e461 更新 README.md 2026-04-27 12:04:29 +08:00
qaiu 45cb255a4d 更新 README.md 2026-04-27 12:01:11 +08:00
140 changed files with 10018 additions and 2789 deletions
+205 -36
View File
@@ -1,24 +1,16 @@
# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time name: Java CIMaven 构建 + Docker 镜像 + 原生环境打包)
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven
# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.
name: Java CI with Maven
# The API requires write permission on the repository to submit dependencies
permissions: permissions:
contents: write contents: write
packages: write packages: write
on: on:
workflow_dispatch:
push: push:
tags: tags:
- '*' # 只有推送tag时才会触发构建 - '*'
branches-ignore: branches-ignore:
- '*' # 排除所有分支的提交 - '*'
paths-ignore: paths-ignore:
- 'bin/**' - 'bin/**'
- '.github/**' - '.github/**'
@@ -32,70 +24,247 @@ on:
- "main" - "main"
jobs: jobs:
# ================================================================
# 阶段一:构建前端 + Maven 打包(只执行一次,产物共享)
# ================================================================
build: build:
name: 编译构建
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - name: 检出代码
uses: actions/checkout@v3
- uses: actions/setup-node@v4 - name: 设置 Node.js 18
uses: actions/setup-node@v4
with: with:
node-version: '18' node-version: '18'
- name: Set up JDK 17 - name: 设置 JDK 17
uses: actions/setup-java@v3 uses: actions/setup-java@v3
with: with:
java-version: '17' java-version: '17'
distribution: 'temurin' distribution: 'temurin'
cache: maven cache: maven
- name: Build Frontend - name: 构建前端
run: cd web-front && yarn install && yarn run build run: cd web-front && yarn install && yarn run build
- name: Build with Maven - name: Maven 编译打包
run: mvn -B package -DskipTests --file pom.xml run: mvn -B package -DskipTests --file pom.xml
# Optional: Uploads the full dependency graph to GitHub to improve the quality of Dependabot alerts this repository can receive - name: 更新依赖图谱
- name: Update dependency graph
uses: advanced-security/maven-dependency-submission-action@v3 uses: advanced-security/maven-dependency-submission-action@v3
if: github.event_name != 'pull_request' if: github.event_name != 'pull_request'
continue-on-error: true
with: with:
ignore-maven-wrapper: true ignore-maven-wrapper: true
# - uses: release-drafter/release-drafter@v5 - name: 分享应用打包目录(供原生包和 Docker 复用)
# env: if: github.event_name != 'pull_request'
# GITHUB_TOKEN: ${{ secrets.ACCESS_TOKEN }}
- name: Upload Artifact
uses: actions/upload-artifact@v4 uses: actions/upload-artifact@v4
with: with:
name: app-package
path: web-service/target/package/
- name: 分享 bin-zip(供 Docker 复用)
if: github.event_name != 'pull_request'
uses: actions/upload-artifact@v4
with:
name: app-bin-zip
path: web-service/target/netdisk-fast-download-bin.zip path: web-service/target/netdisk-fast-download-bin.zip
- name: Login to GitHub Container Registry # ================================================================
# 阶段二-A:Docker 镜像构建(并行)
# ================================================================
docker:
name: Docker 镜像
needs: build
if: github.event_name != 'pull_request' if: github.event_name != 'pull_request'
runs-on: ubuntu-latest
steps:
- name: 检出代码
uses: actions/checkout@v3
- name: 下载 bin-zip 产物
uses: actions/download-artifact@v4
with:
name: app-bin-zip
path: web-service/target/
- name: 登录 GitHub 容器仓库
uses: docker/login-action@v3 uses: docker/login-action@v3
with: with:
registry: ghcr.io registry: ghcr.io
username: ${{ github.actor }} username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }} password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Docker Buildx - name: 设置 QEMU(多平台构建支持)
uses: docker/setup-qemu-action@v3
- name: 设置 Docker Buildx
uses: docker/setup-buildx-action@v3 uses: docker/setup-buildx-action@v3
- name: Extract git tag - name: 生成 Docker 标签
id: tag id: docker_tag
shell: bash
run: | run: |
GIT_TAG=$(git tag --points-at HEAD | head -n 1) tag="$(printf '%s' "${GITHUB_REF_NAME}" | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9_.-]+/-/g; s/^-+//; s/-+$//')"
echo "tag=$GIT_TAG" >> $GITHUB_OUTPUT if [ -z "$tag" ]; then
tag="snapshot"
fi
echo "tag=$tag" >> "$GITHUB_OUTPUT"
- name: Build and push Docker image - name: 构建并推送 Docker 镜像
if: github.event_name != 'pull_request'
uses: docker/build-push-action@v5 uses: docker/build-push-action@v5
with: with:
context: . context: .
push: true push: true
platforms: linux/amd64,linux/arm64,linux/arm/v7 platforms: linux/amd64,linux/arm64,linux/arm/v7
tags: | tags: |
ghcr.io/qaiu/netdisk-fast-download:${{ steps.tag.outputs.tag }} ghcr.io/${{ github.repository }}:${{ steps.docker_tag.outputs.tag }}
ghcr.io/qaiu/netdisk-fast-download:latest ghcr.io/${{ github.repository }}:latest
# ================================================================
# 阶段二-B:原生环境打包 Linux + Windows(并行)
# ================================================================
native-package:
name: 原生环境打包 → ${{ matrix.artifact-name }}
needs: build
if: github.event_name != 'pull_request'
strategy:
matrix:
include:
- os: ubuntu-latest
artifact-name: netdisk-fast-download-linux-amd64
- os: windows-latest
artifact-name: netdisk-fast-download-windows-amd64
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash
steps:
- name: 设置 JDK 17
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'temurin'
- name: 下载 Maven 构建产物
uses: actions/download-artifact@v4
with:
name: app-package
path: web-service/target/package
# ============================================================
# 选择 JDK 模块
# ============================================================
- name: 选择完整 JDK 模块集合
run: |
# 123 网盘/自定义 JS 解析器会通过 javax.script、Nashorn 和 SPI/反射加载模块,
# jdeps 无法稳定识别这些运行时依赖。这里优先保证原生包可运行,使用完整
# JDK 模块集合生成运行时,避免 Windows 精简 JRE 缺 java.scripting/jdk.dynalink 等模块。
printf 'JDK_MODULES=ALL-MODULE-PATH\n' >> $GITHUB_ENV
# ============================================================
# jlink 生成完整模块 JRE
# ============================================================
- name: 生成完整模块 JREjlink
run: |
jlink \
--module-path "$JAVA_HOME/jmods" \
--add-modules "$JDK_MODULES" \
--output "native-package/netdisk-fast-download/jre" \
--strip-debug \
--compress=2 \
--no-header-files \
--no-man-pages
echo "JRE size:"
du -sh native-package/netdisk-fast-download/jre || true
# Windows: 确保 MSVC 运行时 DLL 到位
if [[ "$RUNNER_OS" == "Windows" ]]; then
JRE_BIN="native-package/netdisk-fast-download/jre/bin"
for dll in vcruntime140.dll msvcp140.dll vcruntime140_1.dll; do
if [ ! -f "$JRE_BIN/$dll" ] && [ -f "$JAVA_HOME/bin/$dll" ]; then
echo "jlink 未包含 $dll,从 JDK 补拷"
cp "$JAVA_HOME/bin/$dll" "$JRE_BIN/"
fi
done
echo "=== JRE bin 目录 DLL 清单 ==="
ls -la "$JRE_BIN"/*.dll 2>/dev/null || echo "(无 .dll 文件)"
fi
echo "=== 关键 JDK 模块校验 ==="
"native-package/netdisk-fast-download/jre/bin/java" --list-modules | grep -q '^java.scripting@'
"native-package/netdisk-fast-download/jre/bin/java" --list-modules | grep -q '^jdk.dynalink@'
"native-package/netdisk-fast-download/jre/bin/java" --list-modules | grep -q '^jdk.unsupported@'
# ============================================================
# 组装包目录
# ============================================================
- name: 组装包目录
run: |
PKG="native-package/netdisk-fast-download"
SRC="web-service/target/package"
cp "$SRC/netdisk-fast-download.jar" "$PKG/"
cp -r "$SRC/lib" "$PKG/"
cp -r "$SRC/resources" "$PKG/"
cp -r "$SRC/webroot" "$PKG/"
mkdir -p "$PKG/db"
mkdir -p "$PKG/logs"
# ============================================================
# 生成启动脚本
# ============================================================
- name: 生成启动脚本(Linux
run: |
PKG="native-package/netdisk-fast-download"
echo '#!/bin/bash' > "$PKG/run.sh"
echo 'DIR="$(cd "$(dirname "$0")" && pwd)"' >> "$PKG/run.sh"
echo 'cd "$DIR" || exit 1' >> "$PKG/run.sh"
echo 'exec "$DIR/jre/bin/java" -Xmx512M -Dfile.encoding=utf-8 -jar "$DIR/netdisk-fast-download.jar" "$@"' >> "$PKG/run.sh"
chmod +x "$PKG/run.sh"
- name: 生成启动脚本(Windows
run: |
PKG="native-package/netdisk-fast-download"
echo '@echo off' > "$PKG/run.bat"
echo 'chcp 65001 > nul' >> "$PKG/run.bat"
echo 'pushd %~dp0' >> "$PKG/run.bat"
echo '"%~dp0jre\bin\java.exe" -Xmx512M -Dfile.encoding=utf-8 -jar "%~dp0netdisk-fast-download.jar" %*' >> "$PKG/run.bat"
# ============================================================
# 打包为 zip
# ============================================================
- name: 打包 ZIPLinux
if: runner.os == 'Linux'
run: |
cd native-package
zip -r "../${{ matrix.artifact-name }}.zip" netdisk-fast-download/
- name: 打包 ZIPWindows
if: runner.os == 'Windows'
shell: pwsh
run: |
Compress-Archive -Path native-package/netdisk-fast-download -DestinationPath "${{ matrix.artifact-name }}.zip"
# ============================================================
# 上传产物
# ============================================================
- name: 上传原生安装包
uses: actions/upload-artifact@v4
with:
name: ${{ matrix.artifact-name }}
path: ${{ matrix.artifact-name }}.zip
- name: 上传到 Release
uses: softprops/action-gh-release@v2
with:
files: ${{ matrix.artifact-name }}.zip
tag_name: ${{ github.ref_name }}
generate_release_notes: true
+2
View File
@@ -31,6 +31,7 @@ target/
sdkTest.log sdkTest.log
app.yml app.yml
app-local.yml app-local.yml
secret.yml
#some local files #some local files
@@ -91,3 +92,4 @@ yarn-error.log*
**/${project.build.directory}/ **/${project.build.directory}/
**/${project.basedir}/target/ **/${project.basedir}/target/
**/${basedir}/target/ **/${basedir}/target/
.spec-workflow/
+8 -3
View File
@@ -10,8 +10,13 @@ COPY ./web-service/target/netdisk-fast-download-bin.zip .
RUN unzip netdisk-fast-download-bin.zip && \ RUN unzip netdisk-fast-download-bin.zip && \
mv netdisk-fast-download/* ./ && \ mv netdisk-fast-download/* ./ && \
rm netdisk-fast-download-bin.zip && \ rm netdisk-fast-download-bin.zip && \
chmod +x run.sh chmod +x run.sh && \
mkdir -p db logs
EXPOSE 6400 6401 COPY ./docker-entrypoint.sh /docker-entrypoint.sh
RUN chmod +x /docker-entrypoint.sh
ENTRYPOINT ["sh", "run.sh"] EXPOSE 6401
RUN addgroup --system appgroup && adduser --system --ingroup appgroup appuser
ENTRYPOINT ["/docker-entrypoint.sh"]
+70 -24
View File
@@ -1,30 +1,26 @@
# 一款网盘分享链接云解析快速下载服务 # 一款网盘分享链接云解析快速下载服务
QQ交流群:1017480890 QQ交流群:1017480890
<p align="center"> <p align="center">
<a href="https://github.com/qaiu/netdisk-fast-download/actions/workflows/maven.yml"><img src="https://img.shields.io/github/actions/workflow/status/qaiu/netdisk-fast-download/maven.yml?branch=v0.1.9b8a&style=flat"></a> <a href="https://github.com/qaiu/netdisk-fast-download/actions/workflows/maven.yml"><img src="https://img.shields.io/github/actions/workflow/status/qaiu/netdisk-fast-download/build.yml?branch=main&style=flat"></a>
<a href="https://www.oracle.com/cn/java/technologies/downloads"><img src="https://img.shields.io/badge/jdk-%3E%3D17-blue"></a> <a href="https://www.oracle.com/cn/java/technologies/downloads"><img src="https://img.shields.io/badge/jdk-%3E%3D17-blue"></a>
<a href="https://vertx-china.github.io"><img src="https://img.shields.io/badge/vert.x-4.5.24-blue?style=flat"></a> <a href="https://vertx-china.github.io"><img src="https://img.shields.io/badge/vert.x-4.5.27-blue?style=flat"></a>
<a href="https://raw.githubusercontent.com/qaiu/netdisk-fast-download/master/LICENSE"><img src="https://img.shields.io/github/license/qaiu/netdisk-fast-download?style=flat"></a> <a href="https://raw.githubusercontent.com/qaiu/netdisk-fast-download/master/LICENSE"><img src="https://img.shields.io/github/license/qaiu/netdisk-fast-download?style=flat"></a>
<a href="https://github.com/qaiu/netdisk-fast-download/releases/"><img src="https://img.shields.io/github/v/release/qaiu/netdisk-fast-download?style=flat"></a> <a href="https://github.com/qaiu/netdisk-fast-download/releases/"><img src="https://img.shields.io/github/v/release/qaiu/netdisk-fast-download?style=flat"></a>
<a href="https://oosmetrics.com/repo/qaiu/netdisk-fast-download"><img src="https://api.oosmetrics.com/api/v1/badge/achievement/826aa27a-6e59-4de5-b7fa-cd189f484035.svg"></a> <a href="https://github.com/QAIU/netdisk-fast-download">
<img src="https://img.shields.io/github/stars/QAIU/netdisk-fast-download?style=flat&logo=github" alt="GitHub Stars">
</a>
<a href="https://atomgit.com/QAIU/netdisk-fast-download"><img src="https://atomgit.com/QAIU/netdisk-fast-download/star/badge.svg" alt="AtomGit"></a>
<p align="center"> <p align="center">
<a href="https://trendshift.io/repositories/12101" target="_blank"><img src="https://trendshift.io/api/badge/repositories/12101" alt="qaiu%2Fnetdisk-fast-download | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a> <a href="https://trendshift.io/repositories/12101" target="_blank"><img src="https://trendshift.io/api/badge/repositories/12101" alt="qaiu%2Fnetdisk-fast-download | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p> </p>
<div align="center" style="display:flex; justify-content:center; gap:10px; align-items:flex-start;">
<img
src="https://github.com/user-attachments/assets/bf266d0a-aaf8-4772-9231-e38a4b7bb6cb"
alt="image1"
style="width:300px; max-width:300px; flex:none;"
>
<img
src="https://github.com/user-attachments/assets/bb7a85f0-c256-4b4a-a11b-3ceb55afc302"
alt="image2"
style="width:300px; max-width:300px; flex:none;"
>
</div>
> netdisk-fast-download网盘直链解析可以把云盘分享链接转为直链,可广泛应用于各类下载站,资源站,个人博客,图床,APP下载更新,视频点播等领域。支持市面各大主流云盘的文件分享以及文件夹分享链接,已支持蓝奏云/蓝奏云优享/移动云云空间/小飞机盘/亿方云/123云盘/Cloudreve等,支持加密分享,以及部分网盘文件夹分享。 ![alt text](web-front/img/image.png)
## 介绍
> netdisk-fast-download网盘直链解析可以把云盘分享链接转为直链,可广泛应用于各类下载站,资源站,个人博客,图床,APP下载更新,视频点播等领域。支持市面各大主流云盘的文件分享以及文件夹分享链接,已支持蓝奏云/蓝奏云优享/移动云云空间/小飞机盘/亿方云/123云盘/永硕E盘/Cloudreve等,支持加密分享,以及部分网盘文件夹分享。
[官方文档](https://nfd-parser.github.io/) [官方文档](https://nfd-parser.github.io/)
[API接入](https://nfdparser.apifox.cn/) [API接入](https://nfdparser.apifox.cn/)
@@ -42,12 +38,12 @@ curl -LOJ "https://lz.qaiu.top/parser?url=https://share.feijipan.com/s/Tk1F2kGQ&
```shell ```shell
wget -O bilibili.mp4 "https://lz.qaiu.top/parser?url=https://share.feijipan.com/s/Tk1F2kGQ&pwd=1234" wget -O bilibili.mp4 "https://lz.qaiu.top/parser?url=https://share.feijipan.com/s/Tk1F2kGQ&pwd=1234"
``` ```
或者使用浏览器[直接访问](https://nfd-parser.github.io/nfd-preview/preview.html?src=https%3A%2F%2Flz.qaiu.top%2Fparser%3Furl%3Dhttps%3A%2F%2Fshare.feijipan.com%2Fs%2FTk1F2kGQ&name=bilibili.mp4&ext=mp4): 或者使用浏览器[直接访问](https://nfd-parser.github.io/nfd-preview/preview.html?src=https%3A%2F%2Flz.qaiu.top%2Fparser%3Furl%3Dhttps%3A%2F%2Fwww.ilanzou.com%2Fs%2FCDx6xKbT&name=bilibili.mp4&ext=mp4):
``` ```
### 调用演示站下载: ### 调用演示站下载:
https://lz.qaiu.top/parser?url=https://share.feijipan.com/s/Tk1F2kGQ&pwd=1234 https://lz.qaiu.top/parser?url=https://www.ilanzou.com/s/CDx6xKbT&pwd=1234
### 调用演示站预览: ### 调用演示站预览:
https://nfd-parser.github.io/nfd-preview/preview.html?src=https%3A%2F%2Flz.qaiu.top%2Fparser%3Furl%3Dhttps%3A%2F%2Fshare.feijipan.com%2Fs%2FTk1F2kGQ&name=bilibili.mp4&ext=mp4 https://nfd-parser.github.io/nfd-preview/preview.html?src=https%3A%2F%2Flz.qaiu.top%2Fparser%3Furl%3Dhttps%3A%2F%2Fwww.ilanzou.com%2Fs%2FCDx6xKbT&name=bilibili.mp4&ext=mp4
``` ```
@@ -73,13 +69,14 @@ https://nfd-parser.github.io/nfd-preview/preview.html?src=https%3A%2F%2Flz.qaiu.
- [移动云云空间-ec](https://www.ecpan.cn/web) - [移动云云空间-ec](https://www.ecpan.cn/web)
- [小飞机网盘-fj](https://www.feijipan.com/) - [小飞机网盘-fj](https://www.feijipan.com/)
- [亿方云-fc](https://www.fangcloud.com/) - [亿方云-fc](https://www.fangcloud.com/)
- [123云盘-ye](https://www.123pan.com/) - [123云盘-ye](https://www.123pan.com/) ⚠️仅建议本地部署使用,需登录认证,公共/云端服务器风控严格建议自行部署(Windows 可直接用 run.bat 一键运行)
- ~[115网盘(失效)-p115](https://115.com/)~ - ~[115网盘(失效)-p115](https://115.com/)~
- [文叔叔-ws](https://www.wenshushu.cn/) - [文叔叔-ws](https://www.wenshushu.cn/)
- [联想乐云-le](https://lecloud.lenovo.com/) - [联想乐云-le](https://lecloud.lenovo.com/)
- [QQ邮箱云盘-qqw](https://mail.qq.com/) - [QQ邮箱云盘-qqw](https://mail.qq.com/)
- [QQ闪传-qqsc](https://nutty.qq.com/nutty/ssr/26797.html) - [QQ闪传-qqsc](https://nutty.qq.com/nutty/ssr/26797.html)
- [城通网盘-ct](https://www.ctfile.com) - [城通网盘-ct](https://www.ctfile.com)
- [永硕E盘-ys](https://www.ysepan.com/)(空间分享,如 `https://xxx.ysepan.com/`,密码为空间访问密码;多文件请用文件列表接口)
- [网易云音乐分享链接-mnes](https://music.163.com) - [网易云音乐分享链接-mnes](https://music.163.com)
- [酷狗音乐分享链接-mkgs](https://www.kugou.com) - [酷狗音乐分享链接-mkgs](https://www.kugou.com)
- [酷我音乐分享链接-mkws](https://kuwo.cn) - [酷我音乐分享链接-mkws](https://kuwo.cn)
@@ -95,6 +92,13 @@ https://nfd-parser.github.io/nfd-preview/preview.html?src=https%3A%2F%2Flz.qaiu.
- Onedrive-pod - Onedrive-pod
- Dropbox-pdp - Dropbox-pdp
- iCloud-pic - iCloud-pic
### 专业版提供
- 迅雷网盘-xl
- [夸克云盘-qk](https://pan.quark.cn/)
- [UC云盘-uc](https://fast.uc.cn/)
- [移动云盘-p139](https://yun.139.com/)
- [联通云盘-pwo](https://pan.wo.cn/)
- [天翼云盘-p189](https://cloud.189.cn/)
## API接口 ## API接口
@@ -173,6 +177,7 @@ GET /json/getFileList?url={分享链接}&pwd={密码}
| UC网盘(UC) | **必须** | 必须配置 Cookie 才能解析 | | UC网盘(UC) | **必须** | 必须配置 Cookie 才能解析 |
| 小飞机网盘(FJ) | 可选 | 大文件(>100MB)需要认证 | | 小飞机网盘(FJ) | 可选 | 大文件(>100MB)需要认证 |
| 蓝奏优享(IZ) | 可选 | 大文件需要认证 | | 蓝奏优享(IZ) | 可选 | 大文件需要认证 |
| 123网盘(YE) | 可选 | 需要下载大文件/需要登录的分享时才需要认证,支持账号密码或 token/authorization |
**使用示例** **使用示例**
``` ```
@@ -193,6 +198,38 @@ GET /parser?url={分享链接}&pwd={密码}&auth={加密后的认证参数}
- 目的:防止客户端伪造失败计数请求 - 目的:防止客户端伪造失败计数请求
- 建议:使用高强度随机字符串,且不要与 `authEncryptKey` 相同 - 建议:使用高强度随机字符串,且不要与 `authEncryptKey` 相同
#### `auth` 临时认证参数 与 `auths` 静态配置认证 的区别
本项目存在两种互相独立的认证配置方式,作用范围不同,不要混淆:
| 方式 | 配置位置 | 生效范围 | 适用场景 |
|------|---------|---------|---------|
| `auth` 临时认证参数 | 请求 URL 上的 `auth` 查询参数 | **仅当次请求**,优先级高于 app-dev.yml 中的静态配置 | 调用方按用户临时提供的账号/Cookie/token 解析,不同请求可携带不同认证信息 |
| `auths` 静态配置认证 | `app-dev.yml``auths.<网盘标识>` 节点 | **服务端长期生效**,所有未携带 `auth` 参数的请求都会复用 | 部署方自己长期配置一份账号,供所有请求默认使用 |
以 123网盘(`ye`)为例,`app-dev.yml` 中支持以下几种写法(三选一即可):
```yaml
auths:
ye:
username: 你的123网盘账号
password: 你的123网盘密码
```
```yaml
auths:
ye:
token: 已登录后获取的 Authorization/AccessToken
```
```yaml
auths:
ye:
authorization: 已登录后获取的 Authorization/AccessToken # 与 token 等价,二选一
```
> ⚠️ 注意:YAML 中 key 后面不写值(如 `authorization:` 空着)等同于没配置,不会生效,必须填入真实的账号密码或 token 内容。
如果只是临时调用一次,不想改动服务端配置,也可以用上面提到的 `auth` 参数临时传递(`authType` 可选 `password`/`accesstoken`/`authorization`),无需重启服务,仅本次请求生效。
### 特殊说明 ### 特殊说明
- 移动云云空间的 `分享key` 取分享链接中的 `data` 参数值 - 移动云云空间的 `分享key` 取分享链接中的 `data` 参数值
@@ -273,9 +310,12 @@ json返回数据格式示例:
"timestamp": 1736489219402 "timestamp": 1736489219402
} }
``` ```
#### 3. 文件夹解析(支持蓝奏云/蓝奏优享/小飞机网盘) #### 3. 文件夹解析(支持蓝奏云/蓝奏优享/小飞机/永硕E盘等)
/v2/getFileList?url=分享链接&pwd=分享密码 /v2/getFileList?url=分享链接&pwd=分享密码
永硕E盘(`ys`)空间链接先返回目录列表,再带 `dirId` 获取目录内文件:
`/v2/getFileList?url=https://xxx.ysepan.com/&pwd=空间密码&dirId=目录编号`
```json ```json
{ {
"code": 200, "code": 200,
@@ -331,6 +371,7 @@ json返回数据格式示例:
| 360亿方云 | √ | √ | 100G(须实名) | 不限大小 | | 360亿方云 | √ | √ | 100G(须实名) | 不限大小 |
| 123云盘 | √ | √ | 2T | 100G>100M需要登录) | | 123云盘 | √ | √ | 2T | 100G>100M需要登录) |
| 文叔叔 | √ | √ | 10G | 5GB | | 文叔叔 | √ | √ | 10G | 5GB |
| 永硕E盘 | √ | √(空间密码) | 视套餐 | 视套餐 |
| WPS云文档 | √ | X | 5G(免费) | 10M(免费)/2G(会员) | | WPS云文档 | √ | X | 5G(免费) | 10M(免费)/2G(会员) |
| 夸克网盘 | x | √ | 10G | 不限大小 | | 夸克网盘 | x | √ | 10G | 不限大小 |
| UC网盘 | x | √ | 10G | 不限大小 | | UC网盘 | x | √ | 10G | 不限大小 |
@@ -414,7 +455,7 @@ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtow
> 注意: netdisk-fast-download.service中的ExecStart的路径改为实际路径 > 注意: netdisk-fast-download.service中的ExecStart的路径改为实际路径
```shell ```shell
cd ~ cd ~
wget -O netdisk-fast-download.zip https://github.com/qaiu/netdisk-fast-download/releases/download/v0.1.9b7/netdisk-fast-download-bin.zip wget -O netdisk-fast-download.zip https://github.com/qaiu/netdisk-fast-download/releases/download/v3.0.2/netdisk-fast-download-bin.zip
unzip netdisk-fast-download-bin.zip unzip netdisk-fast-download-bin.zip
cd netdisk-fast-download cd netdisk-fast-download
bash service-install.sh bash service-install.sh
@@ -495,7 +536,13 @@ Core模块集成Vert.x实现类似spring的注解式路由API
## Star History ## Star History
[![Star History Chart](https://api.star-history.com/svg?repos=qaiu/netdisk-fast-download&type=Date)](https://star-history.com/#qaiu/netdisk-fast-download&Date) <a href="https://www.star-history.com/?repos=qaiu%2Fnetdisk-fast-download&type=date&legend=bottom-right">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=qaiu/netdisk-fast-download&type=date&theme=dark&legend=bottom-right&sealed_token=dfQO_dJcTqcPkEnM7SfxRyHoFbV5Ah4LxoEhdlheMn4T2YLEV_WETxFZexeAbWN5OmNyYuycWan2d42PAFbw0CuU4oCTKgehfErFJ9eVl2CyVpP_4xrdQw" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=qaiu/netdisk-fast-download&type=date&legend=bottom-right&sealed_token=dfQO_dJcTqcPkEnM7SfxRyHoFbV5Ah4LxoEhdlheMn4T2YLEV_WETxFZexeAbWN5OmNyYuycWan2d42PAFbw0CuU4oCTKgehfErFJ9eVl2CyVpP_4xrdQw" />
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=qaiu/netdisk-fast-download&type=date&legend=bottom-right&sealed_token=dfQO_dJcTqcPkEnM7SfxRyHoFbV5Ah4LxoEhdlheMn4T2YLEV_WETxFZexeAbWN5OmNyYuycWan2d42PAFbw0CuU4oCTKgehfErFJ9eVl2CyVpP_4xrdQw" />
</picture>
</a>
## **免责声明** ## **免责声明**
- 用户在使用本项目时,应自行承担风险,并确保其行为符合当地法律法规。开发者不对用户因使用本项目而导致的任何后果负责。 - 用户在使用本项目时,应自行承担风险,并确保其行为符合当地法律法规。开发者不对用户因使用本项目而导致的任何后果负责。
@@ -513,4 +560,3 @@ Core模块集成Vert.x实现类似spring的注解式路由API
+1 -2
View File
@@ -1,6 +1,5 @@
#!/bin/bash #!/bin/bash
# set -x # set -x
LAUNCH_JAR="netdisk-fast-download.jar" LAUNCH_JAR="netdisk-fast-download.jar"
nohup java -Xmx512M -jar "$LAUNCH_JAR" "$@" >startup.log 2>&1 & exec java -Xmx${JVM_XMX:-512M} ${JVM_OPTS} -jar "$LAUNCH_JAR" "$@"
tail -f startup.log
+1 -1
View File
@@ -65,7 +65,7 @@
<dependency> <dependency>
<groupId>org.postgresql</groupId> <groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId> <artifactId>postgresql</artifactId>
<version>42.7.3</version> <version>42.7.13</version>
</dependency> </dependency>
</dependencies> </dependencies>
@@ -53,7 +53,7 @@ public class CreateDatabase {
stmt.executeUpdate("CREATE DATABASE IF NOT EXISTS " + dbName + " CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci"); stmt.executeUpdate("CREATE DATABASE IF NOT EXISTS " + dbName + " CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci");
LOGGER.info(">>>>>>>>>>> 数据库'{}'创建成功 <<<<<<<<<<<<", dbName); LOGGER.info(">>>>>>>>>>> 数据库'{}'创建成功 <<<<<<<<<<<<", dbName);
} catch (SQLException e) { } catch (SQLException e) {
e.printStackTrace(); LOGGER.error("创建数据库失败", e);
} }
} }
@@ -24,35 +24,39 @@ import java.util.*;
* @author <a href="https://qaiu.top">QAIU</a> * @author <a href="https://qaiu.top">QAIU</a>
*/ */
public class CreateTable { public class CreateTable {
public static Map<Class<?>, String> javaProperty2SqlColumnMap = new HashMap<>() {{ public static final Map<Class<?>, String> javaProperty2SqlColumnMap;
static {
Map<Class<?>, String> map = new HashMap<>();
// Java类型到SQL类型的映射 // Java类型到SQL类型的映射
put(Integer.class, "INT"); map.put(Integer.class, "INT");
put(Short.class, "SMALLINT"); map.put(Short.class, "SMALLINT");
put(Byte.class, "TINYINT"); map.put(Byte.class, "TINYINT");
put(Long.class, "BIGINT"); map.put(Long.class, "BIGINT");
put(java.math.BigDecimal.class, "DECIMAL"); map.put(java.math.BigDecimal.class, "DECIMAL");
put(Double.class, "DOUBLE"); map.put(Double.class, "DOUBLE");
put(Float.class, "REAL"); map.put(Float.class, "REAL");
put(Boolean.class, "BOOLEAN"); map.put(Boolean.class, "BOOLEAN");
put(String.class, "VARCHAR"); map.put(String.class, "VARCHAR");
put(Date.class, "TIMESTAMP"); map.put(Date.class, "TIMESTAMP");
put(java.time.LocalDateTime.class, "TIMESTAMP"); map.put(java.time.LocalDateTime.class, "TIMESTAMP");
put(java.sql.Timestamp.class, "TIMESTAMP"); map.put(java.sql.Timestamp.class, "TIMESTAMP");
put(java.sql.Date.class, "DATE"); map.put(java.sql.Date.class, "DATE");
put(java.sql.Time.class, "TIME"); map.put(java.sql.Time.class, "TIME");
// 基本数据类型 // 基本数据类型
put(int.class, "INT"); map.put(int.class, "INT");
put(short.class, "SMALLINT"); map.put(short.class, "SMALLINT");
put(byte.class, "TINYINT"); map.put(byte.class, "TINYINT");
put(long.class, "BIGINT"); map.put(long.class, "BIGINT");
put(double.class, "DOUBLE"); map.put(double.class, "DOUBLE");
put(float.class, "REAL"); map.put(float.class, "REAL");
put(boolean.class, "BOOLEAN"); map.put(boolean.class, "BOOLEAN");
}};
javaProperty2SqlColumnMap = Collections.unmodifiableMap(map);
}
private static final Logger LOGGER = LoggerFactory.getLogger(CreateTable.class); private static final Logger LOGGER = LoggerFactory.getLogger(CreateTable.class);
public static String UNIQUE_PREFIX = "idx_"; public static final String UNIQUE_PREFIX = "idx_";
private static Case getCase(Class<?> clz) { private static Case getCase(Class<?> clz) {
return switch (clz.getName()) { return switch (clz.getName()) {
@@ -17,7 +17,7 @@ import org.slf4j.LoggerFactory;
* *
* @author <a href="https://qaiu.top">QAIU</a> * @author <a href="https://qaiu.top">QAIU</a>
*/ */
public class JDBCPoolInit { public class JDBCPoolInit implements AutoCloseable {
private static final Logger LOGGER = LoggerFactory.getLogger(JDBCPoolInit.class); private static final Logger LOGGER = LoggerFactory.getLogger(JDBCPoolInit.class);
@@ -101,4 +101,16 @@ public class JDBCPoolInit {
synchronized public JDBCPool getPool() { synchronized public JDBCPool getPool() {
return pool; return pool;
} }
/**
* 关闭连接池,释放数据库资源
*/
@Override
public synchronized void close() {
if (pool != null) {
pool.close();
LOGGER.info("数据库连接池已关闭: URL={}", url);
pool = null;
}
}
} }
+78 -10
View File
@@ -16,9 +16,13 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
import java.lang.management.ManagementFactory; import java.lang.management.ManagementFactory;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Calendar; import java.util.Calendar;
import java.util.Date; import java.util.Date;
import java.util.UUID; import java.util.UUID;
import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.locks.LockSupport; import java.util.concurrent.locks.LockSupport;
import static cn.qaiu.vx.core.util.ConfigConstant.*; import static cn.qaiu.vx.core.util.ConfigConstant.*;
@@ -43,11 +47,25 @@ public final class Deploy {
private Handler<JsonObject> handle; private Handler<JsonObject> handle;
private Thread mainThread; private Thread mainThread;
private final List<Runnable> preShutdownTasks = new CopyOnWriteArrayList<>();
private final List<Runnable> postShutdownTasks = new CopyOnWriteArrayList<>();
public static Deploy instance() { public static Deploy instance() {
return INSTANCE; return INSTANCE;
} }
public void addPreShutdownTask(Runnable task) {
if (task != null) {
preShutdownTasks.add(task);
}
}
public void addPostShutdownTask(Runnable task) {
if (task != null) {
postShutdownTasks.add(task);
}
}
/** /**
* *
* @param args 启动参数 * @param args 启动参数
@@ -62,10 +80,19 @@ public final class Deploy {
path.append("-").append(args[0].replace("app-","")); path.append("-").append(args[0].replace("app-",""));
} }
// 读取yml配置 // 读取yml配置,优先当前目录,其次 resources/ 子目录
String configFile = path + ".yml";
if (!Files.exists(Path.of(configFile)) && Files.exists(Path.of("resources", configFile))) {
path.insert(0, "resources/");
LOGGER.info("从 resources/ 目录加载配置: {}", path + ".yml");
}
ConfigUtil.readYamlConfig(path.toString(), tempVertx) ConfigUtil.readYamlConfig(path.toString(), tempVertx)
.onSuccess(this::readConf) .onSuccess(this::readConf)
.onFailure(Throwable::printStackTrace); .onFailure(err -> {
LOGGER.error("读取配置文件失败: {}", err.getMessage(), err);
LockSupport.unpark(mainThread);
System.exit(-1);
});
LockSupport.park(); LockSupport.park();
deployVerticle(); deployVerticle();
} }
@@ -122,9 +149,16 @@ public final class Deploy {
customConfig = globalConfig.getJsonObject(CUSTOM); customConfig = globalConfig.getJsonObject(CUSTOM);
JsonObject vertxConfig = globalConfig.getJsonObject(VERTX); JsonObject vertxConfig = globalConfig.getJsonObject(VERTX);
Integer vertxConfigELPS = vertxConfig.getInteger(EVENT_LOOP_POOL_SIZE); JsonObject vertxOptionsConfig = vertxConfig.copy();
var vertxOptions = vertxConfigELPS == 0 ? if (vertxOptionsConfig.getInteger(EVENT_LOOP_POOL_SIZE, 0) == 0) {
new VertxOptions() : new VertxOptions(vertxConfig); vertxOptionsConfig.remove(EVENT_LOOP_POOL_SIZE);
}
if (vertxOptionsConfig.getInteger("workerPoolSize", 0) == 0) {
vertxOptionsConfig.remove("workerPoolSize");
}
Integer vertxConfigELPS = vertxConfig.getInteger(EVENT_LOOP_POOL_SIZE, 0);
var vertxOptions = vertxOptionsConfig.isEmpty() ?
new VertxOptions() : new VertxOptions(vertxOptionsConfig);
// vertxOptions.setAddressResolverOptions( // vertxOptions.setAddressResolverOptions(
// new AddressResolverOptions(). // new AddressResolverOptions().
@@ -137,30 +171,46 @@ public final class Deploy {
vertxOptions.getWorkerPoolSize()); vertxOptions.getWorkerPoolSize());
var vertx = Vertx.vertx(vertxOptions); var vertx = Vertx.vertx(vertxOptions);
VertxHolder.init(vertx); VertxHolder.init(vertx);
// 注册 ShutdownHook,确保进程退出时优雅关闭资源
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
LOGGER.info("JVM shutting down...");
runShutdownTasks("before Vert.x close", preShutdownTasks);
try {
LOGGER.info("Closing Vert.x...");
vertx.close().toCompletionStage().toCompletableFuture().get(10, java.util.concurrent.TimeUnit.SECONDS);
LOGGER.info("Vert.x closed successfully");
} catch (Exception e) {
LOGGER.warn("Vert.x close error or timeout", e);
} finally {
runShutdownTasks("after Vert.x close", postShutdownTasks);
}
}));
//配置保存在共享数据中 //配置保存在共享数据中
var sharedData = vertx.sharedData(); var sharedData = vertx.sharedData();
LocalMap<String, Object> localMap = sharedData.getLocalMap(LOCAL); LocalMap<String, Object> localMap = sharedData.getLocalMap(LOCAL);
localMap.put(GLOBAL_CONFIG, globalConfig); localMap.put(GLOBAL_CONFIG, globalConfig);
localMap.put(CUSTOM_CONFIG, customConfig); localMap.put(CUSTOM_CONFIG, customConfig);
localMap.put(SERVER, globalConfig.getJsonObject(SERVER)); localMap.put(SERVER, globalConfig.getJsonObject(SERVER));
var future0 = vertx.createSharedWorkerExecutor("other-handle") WorkerExecutor otherHandleExecutor = vertx.createSharedWorkerExecutor("other-handle");
.executeBlocking(() -> { var future0 = otherHandleExecutor.executeBlocking(() -> {
handle.handle(globalConfig); handle.handle(globalConfig);
return "Other handle complete"; return "Other handle complete";
}); });
future0.onSuccess(res -> { future0.onSuccess(res -> {
otherHandleExecutor.close();
LOGGER.info(res); LOGGER.info(res);
// 部署 路由、异步service、反向代理 服务 // 部署 路由、异步service、反向代理 服务
var future1 = vertx.deployVerticle(RouterVerticle.class, getWorkDeploymentOptions("Router")); var future1 = vertx.deployVerticle(RouterVerticle.class, getWorkDeploymentOptions("Router"));
var future2 = vertx.deployVerticle(ServiceVerticle.class, getWorkDeploymentOptions("Service")); var future2 = vertx.deployVerticle(ServiceVerticle.class, getWorkDeploymentOptions("Service"));
var future3 = vertx.deployVerticle(ReverseProxyVerticle.class, getWorkDeploymentOptions("proxy")); var future3 = vertx.deployVerticle(ReverseProxyVerticle.class, getWorkDeploymentOptions("proxy", 1));
JsonObject jsonObject = ((JsonObject) localMap.get(GLOBAL_CONFIG)).getJsonObject("proxy-server"); JsonObject jsonObject = ((JsonObject) localMap.get(GLOBAL_CONFIG)).getJsonObject("proxy-server");
if (jsonObject != null) { if (jsonObject != null) {
genPwd(jsonObject); genPwd(jsonObject);
var future4 = vertx.deployVerticle(HttpProxyVerticle.class, getWorkDeploymentOptions("proxy")); var future4 = vertx.deployVerticle(HttpProxyVerticle.class, getWorkDeploymentOptions("proxy", 1));
future4.onSuccess(LOGGER::info); future4.onSuccess(LOGGER::info);
future4.onFailure(e -> LOGGER.error("Other handle error", e)); future4.onFailure(e -> LOGGER.error("Other handle error", e));
Future.all(future1, future2, future3, future4) Future.all(future1, future2, future3, future4)
@@ -172,7 +222,10 @@ public final class Deploy {
.onFailure(this::deployVerticalFailed); .onFailure(this::deployVerticalFailed);
} }
}).onFailure(e -> LOGGER.error("Other handle error", e)); }).onFailure(e -> {
otherHandleExecutor.close();
LOGGER.error("Other handle error", e);
});
} }
private static void genPwd(JsonObject jsonObject) { private static void genPwd(JsonObject jsonObject) {
@@ -189,6 +242,21 @@ public final class Deploy {
jsonObject.getString("password")); jsonObject.getString("password"));
LOGGER.info("==============server info================"); LOGGER.info("==============server info================");
} }
private static void runShutdownTasks(String stage, List<Runnable> tasks) {
if (tasks.isEmpty()) {
return;
}
LOGGER.info("Running {} shutdown tasks: {}", stage, tasks.size());
for (Runnable task : tasks) {
try {
task.run();
} catch (Exception e) {
LOGGER.warn("Shutdown task failed at stage {}", stage, e);
}
}
}
/** /**
* 部署失败 * 部署失败
* *
@@ -1,14 +1,20 @@
package cn.qaiu.vx.core.base; package cn.qaiu.vx.core.base;
import cn.qaiu.vx.core.annotaions.HandleSortFilter;
import cn.qaiu.vx.core.interceptor.AfterInterceptor; import cn.qaiu.vx.core.interceptor.AfterInterceptor;
import cn.qaiu.vx.core.model.JsonResult; import cn.qaiu.vx.core.model.JsonResult;
import cn.qaiu.vx.core.util.CommonUtil;
import cn.qaiu.vx.core.util.ReflectionUtil; import cn.qaiu.vx.core.util.ReflectionUtil;
import io.vertx.core.json.JsonObject; import io.vertx.core.json.JsonObject;
import io.vertx.ext.web.RoutingContext; import io.vertx.ext.web.RoutingContext;
import org.reflections.Reflections; import org.reflections.Reflections;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Collections;
import java.util.LinkedHashSet;
import java.util.Objects;
import java.util.Set; import java.util.Set;
import java.util.stream.Collectors;
import static cn.qaiu.vx.core.util.ResponseUtil.*; import static cn.qaiu.vx.core.util.ResponseUtil.*;
@@ -22,9 +28,10 @@ public interface BaseHttpApi {
// 需要扫描注册的Router路径 // 需要扫描注册的Router路径
Reflections reflections = ReflectionUtil.getReflections(); Reflections reflections = ReflectionUtil.getReflections();
Logger LOGGER = LoggerFactory.getLogger(BaseHttpApi.class);
default void doFireJsonObjectResponse(RoutingContext ctx, JsonObject jsonObject) { default void doFireJsonObjectResponse(RoutingContext ctx, JsonObject jsonObject) {
if (!ctx.response().ended()) { if (!isResponseDone(ctx)) {
fireJsonObjectResponse(ctx, jsonObject); fireJsonObjectResponse(ctx, jsonObject);
} }
handleAfterInterceptor(ctx, jsonObject); handleAfterInterceptor(ctx, jsonObject);
@@ -32,14 +39,14 @@ public interface BaseHttpApi {
default <T> void doFireJsonResultResponse(RoutingContext ctx, JsonResult<T> jsonResult) { default <T> void doFireJsonResultResponse(RoutingContext ctx, JsonResult<T> jsonResult) {
if (!ctx.response().ended()) { if (!isResponseDone(ctx)) {
fireJsonResultResponse(ctx, jsonResult); fireJsonResultResponse(ctx, jsonResult);
} }
handleAfterInterceptor(ctx, jsonResult.toJsonObject()); handleAfterInterceptor(ctx, jsonResult.toJsonObject());
} }
default void doFireJsonObjectResponse(RoutingContext ctx, JsonObject jsonObject, int statusCode) { default void doFireJsonObjectResponse(RoutingContext ctx, JsonObject jsonObject, int statusCode) {
if (!ctx.response().ended()) { if (!isResponseDone(ctx)) {
fireJsonObjectResponse(ctx, jsonObject, statusCode); fireJsonObjectResponse(ctx, jsonObject, statusCode);
} }
handleAfterInterceptor(ctx, jsonObject); handleAfterInterceptor(ctx, jsonObject);
@@ -47,30 +54,78 @@ public interface BaseHttpApi {
default <T> void doFireJsonResultResponse(RoutingContext ctx, JsonResult<T> jsonResult, int statusCode) { default <T> void doFireJsonResultResponse(RoutingContext ctx, JsonResult<T> jsonResult, int statusCode) {
if (!ctx.response().ended()) { if (!isResponseDone(ctx)) {
fireJsonResultResponse(ctx, jsonResult, statusCode); fireJsonResultResponse(ctx, jsonResult, statusCode);
} }
handleAfterInterceptor(ctx, jsonResult.toJsonObject()); handleAfterInterceptor(ctx, jsonResult.toJsonObject());
} }
default Set<AfterInterceptor> getAfterInterceptor() { default Set<AfterInterceptor> getAfterInterceptor() {
return AfterInterceptorHolder.INSTANCES;
}
class AfterInterceptorHolder {
private static final Set<AfterInterceptor> INSTANCES = loadAfterInterceptors();
private static Set<AfterInterceptor> loadAfterInterceptors() {
Set<Class<? extends AfterInterceptor>> afterInterceptorClassSet = Set<Class<? extends AfterInterceptor>> afterInterceptorClassSet =
reflections.getSubTypesOf(AfterInterceptor.class); reflections.getSubTypesOf(AfterInterceptor.class);
if (afterInterceptorClassSet == null) { if (afterInterceptorClassSet == null || afterInterceptorClassSet.isEmpty()) {
return Collections.emptySet();
}
return afterInterceptorClassSet.stream()
.filter(AfterInterceptorHolder::isEnabled)
.sorted(AfterInterceptorHolder::compareOrder)
.map(AfterInterceptorHolder::newInterceptor)
.filter(Objects::nonNull)
.collect(Collectors.collectingAndThen(
Collectors.toCollection(LinkedHashSet::new),
Collections::unmodifiableSet));
}
private static boolean isEnabled(Class<? extends AfterInterceptor> clazz) {
HandleSortFilter sort = clazz.getAnnotation(HandleSortFilter.class);
return sort == null || sort.value() >= 0;
}
private static int compareOrder(Class<? extends AfterInterceptor> left, Class<? extends AfterInterceptor> right) {
return Integer.compare(order(left), order(right));
}
private static int order(Class<? extends AfterInterceptor> clazz) {
HandleSortFilter sort = clazz.getAnnotation(HandleSortFilter.class);
return sort == null ? 0 : sort.value();
}
private static AfterInterceptor newInterceptor(Class<? extends AfterInterceptor> clazz) {
try {
return ReflectionUtil.newWithNoParam(clazz);
} catch (Exception e) {
LOGGER.warn("AfterInterceptor 初始化失败,已跳过: {}", clazz.getName(), e);
return null; return null;
} }
return CommonUtil.sortClassSet(afterInterceptorClassSet); }
} }
default void handleAfterInterceptor(RoutingContext ctx, JsonObject jsonObject) { default void handleAfterInterceptor(RoutingContext ctx, JsonObject jsonObject) {
Set<AfterInterceptor> afterInterceptor = getAfterInterceptor(); if (ctx.response().closed()) {
if (afterInterceptor != null) { return;
afterInterceptor.forEach(ai -> ai.handle(ctx, jsonObject));
} }
if (!ctx.response().ended()) { Set<AfterInterceptor> afterInterceptor = getAfterInterceptor();
afterInterceptor.forEach(ai -> {
try {
ai.handle(ctx, jsonObject);
} catch (Exception e) {
LOGGER.warn("AfterInterceptor 执行失败: {}", ai.getClass().getName(), e);
}
});
if (!isResponseDone(ctx)) {
fireTextResponse(ctx, "handleAfterInterceptor: response not end"); fireTextResponse(ctx, "handleAfterInterceptor: response not end");
} }
} }
default boolean isResponseDone(RoutingContext ctx) {
return ctx.response().ended() || ctx.response().closed();
}
} }
@@ -34,6 +34,7 @@ import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method; import java.lang.reflect.Method;
import java.time.LocalDateTime; import java.time.LocalDateTime;
import java.util.*; import java.util.*;
import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import java.util.stream.Stream; import java.util.stream.Stream;
@@ -96,7 +97,9 @@ public class RouterHandlerFactory implements BaseHttpApi {
mainRouter.route().handler(CorsHandler.create().addRelativeOrigin(".*").allowCredentials(true).allowedMethods(httpMethods)); mainRouter.route().handler(CorsHandler.create().addRelativeOrigin(".*").allowCredentials(true).allowedMethods(httpMethods));
// 配置文件上传路径 // 配置文件上传路径
mainRouter.route().handler(BodyHandler.create().setUploadsDirectory("uploads")); mainRouter.route().handler(BodyHandler.create()
.setUploadsDirectory("uploads")
.setBodyLimit(2L * 1024 * 1024));
// 拦截器 // 拦截器
Set<Handler<RoutingContext>> interceptorSet = getInterceptorSet(); Set<Handler<RoutingContext>> interceptorSet = getInterceptorSet();
@@ -127,8 +130,9 @@ public class RouterHandlerFactory implements BaseHttpApi {
// 错误请求处理 // 错误请求处理
mainRouter.errorHandler(405, ctx -> doFireJsonResultResponse(ctx, JsonResult mainRouter.errorHandler(405, ctx -> doFireJsonResultResponse(ctx, JsonResult
.error("Method Not Allowed", 405))); .error("Method Not Allowed", 405)));
mainRouter.errorHandler(404, ctx -> ctx.response().setStatusCode(404).setChunked(true) mainRouter.errorHandler(404, ctx -> {
.end("Internal server error: 404 not found")); ctx.response().setStatusCode(404).end("404 not found");
});
return mainRouter; return mainRouter;
} }
@@ -174,13 +178,14 @@ public class RouterHandlerFactory implements BaseHttpApi {
route.handler(TimeoutHandler.create(SharedDataUtil.getCustomConfig().getInteger(ROUTE_TIME_OUT))); route.handler(TimeoutHandler.create(SharedDataUtil.getCustomConfig().getInteger(ROUTE_TIME_OUT)));
route.handler(ResponseTimeHandler.create()); route.handler(ResponseTimeHandler.create());
route.handler(ctx -> handlerMethod(instance, method, ctx)).failureHandler(ctx -> { route.handler(ctx -> handlerMethod(instance, method, ctx)).failureHandler(ctx -> {
if (ctx.response().ended()) return; if (isResponseDone(ctx)) return;
// 超时处理器状态码503 // 超时处理器状态码503
if (ctx.statusCode() == 503 || ctx.failure() == null) { if (ctx.statusCode() == 503 || ctx.failure() == null) {
doFireJsonResultResponse(ctx, JsonResult.error("未知异常, 请联系管理员"), 503); doFireJsonResultResponse(ctx, JsonResult.error("未知异常, 请联系管理员"), 503);
} else { } else {
ctx.failure().printStackTrace(); LOGGER.error("路由处理失败", ctx.failure());
doFireJsonResultResponse(ctx, JsonResult.error(ctx.failure().getMessage()), 500); String msg = ctx.failure() != null ? ctx.failure().getMessage() : "未知异常";
doFireJsonResultResponse(ctx, JsonResult.error(msg), 500);
} }
}); });
} else if (method.isAnnotationPresent(SockRouteMapper.class)) { } else if (method.isAnnotationPresent(SockRouteMapper.class)) {
@@ -198,7 +203,7 @@ public class RouterHandlerFactory implements BaseHttpApi {
try { try {
ReflectionUtil.invokeWithArguments(method, instance, sock); ReflectionUtil.invokeWithArguments(method, instance, sock);
} catch (Throwable e) { } catch (Throwable e) {
e.printStackTrace(); LOGGER.error("WebSocket处理异常", e);
} }
}); });
if (url.endsWith("*")) { if (url.endsWith("*")) {
@@ -322,7 +327,7 @@ public class RouterHandlerFactory implements BaseHttpApi {
parameterValueList.put(k, entity); parameterValueList.put(k, entity);
} }
} catch (ClassNotFoundException e) { } catch (ClassNotFoundException e) {
e.printStackTrace(); LOGGER.error("实体类绑定异常: {}", typeName, e);
} }
} }
}); });
@@ -365,7 +370,7 @@ public class RouterHandlerFactory implements BaseHttpApi {
Object entity = ParamUtil.multiMapToEntity(queryParams, aClass); Object entity = ParamUtil.multiMapToEntity(queryParams, aClass);
parameterValueList.put(k, entity); parameterValueList.put(k, entity);
} catch (Exception e) { } catch (Exception e) {
e.printStackTrace(); LOGGER.error("参数绑定异常: {}", v.getRight().getName(), e);
} }
} else if (parameterValueList.get(k) == null } else if (parameterValueList.get(k) == null
&& JsonObject.class.getName().equals(v.getRight().getName())) { && JsonObject.class.getName().equals(v.getRight().getName())) {
@@ -392,38 +397,43 @@ public class RouterHandlerFactory implements BaseHttpApi {
if (data instanceof JsonResult jsonResult) { if (data instanceof JsonResult jsonResult) {
doFireJsonResultResponse(ctx, (JsonResult<?>) data, jsonResult.getCode()); doFireJsonResultResponse(ctx, (JsonResult<?>) data, jsonResult.getCode());
} } else if (data instanceof JsonObject) {
if (data instanceof JsonObject) {
doFireJsonObjectResponse(ctx, ((JsonObject) data)); doFireJsonObjectResponse(ctx, ((JsonObject) data));
} else if (data instanceof Future) { // 处理异步响应 } else if (data instanceof Future) { // 处理异步响应
((Future<?>) data).onSuccess(res -> { Future<?> responseFuture = (Future<?>) data;
AtomicReference<RoutingContext> ctxRef = new AtomicReference<>(ctx);
ctx.addEndHandler(v -> ctxRef.set(null));
responseFuture.onComplete(ar -> {
RoutingContext responseCtx = ctxRef.getAndSet(null);
if (responseCtx == null || isResponseDone(responseCtx)) {
return;
}
if (ar.succeeded()) {
Object res = ar.result();
if (res instanceof JsonResult jsonResult) { if (res instanceof JsonResult jsonResult) {
doFireJsonResultResponse(ctx, jsonResult, jsonResult.getCode()); doFireJsonResultResponse(responseCtx, jsonResult, jsonResult.getCode());
} } else if (res instanceof JsonObject) {
if (res instanceof JsonObject) { doFireJsonObjectResponse(responseCtx, ((JsonObject) res));
doFireJsonObjectResponse(ctx, ((JsonObject) res));
} else if (res != null) { } else if (res != null) {
doFireJsonResultResponse(ctx, JsonResult.data(res)); doFireJsonResultResponse(responseCtx, JsonResult.data(res));
} else { } else {
doFireJsonResultResponse(ctx, JsonResult.data(null)); doFireJsonResultResponse(responseCtx, JsonResult.data(null));
} }
} else {
}).onFailure(e -> doFireJsonResultResponse(ctx, JsonResult.error(e.getMessage()), 500)); Throwable e = ar.cause();
LOGGER.error("请求处理失败", e);
String msg = e != null && e.getMessage() != null ? e.getMessage() : "服务器内部错误";
doFireJsonResultResponse(responseCtx, JsonResult.error(msg), 500);
}
});
} else { } else {
doFireJsonResultResponse(ctx, JsonResult.data(data)); doFireJsonResultResponse(ctx, JsonResult.data(data));
} }
} }
} catch (Throwable e) { } catch (Throwable e) {
e.printStackTrace(); LOGGER.error("请求处理异常", e);
String err = e.getMessage(); String msg = e.getMessage() != null ? e.getMessage() : "服务器内部错误";
if (e.getCause() != null) { doFireJsonResultResponse(ctx, JsonResult.error(msg), 500);
if (e.getCause() instanceof InvocationTargetException) {
err = ((InvocationTargetException) e.getCause()).getTargetException().getMessage();
} else {
err = e.getCause().getMessage();
}
}
doFireJsonResultResponse(ctx, JsonResult.error(err), 500);
} }
} }
@@ -17,6 +17,8 @@ import java.util.LinkedHashSet;
import java.util.Map; import java.util.Map;
import java.util.Properties; import java.util.Properties;
import java.util.Set; import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.regex.Pattern;
import java.util.stream.Collectors; import java.util.stream.Collectors;
/** /**
@@ -29,6 +31,16 @@ public class CommonUtil {
private static final Logger LOGGER = LoggerFactory.getLogger(CommonUtil.class); private static final Logger LOGGER = LoggerFactory.getLogger(CommonUtil.class);
/** 正则表达式缓存,避免每次调用重新编译 */
private static final ConcurrentHashMap<String, Pattern> PATTERN_CACHE = new ConcurrentHashMap<>();
/**
* 获取预编译的 Pattern(带缓存)
*/
private static Pattern getCachedPattern(String regex) {
return PATTERN_CACHE.computeIfAbsent(regex, Pattern::compile);
}
/** /**
* 匹配正则list * 匹配正则list
* *
@@ -39,7 +51,7 @@ public class CommonUtil {
public static boolean matchRegList(List<?> regList, String destStr) { public static boolean matchRegList(List<?> regList, String destStr) {
// 判断是否忽略 // 判断是否忽略
for (Object ignores : regList) { for (Object ignores : regList) {
if (destStr.matches(ignores.toString())) { if (getCachedPattern(ignores.toString()).matcher(destStr).matches()) {
return true; return true;
} }
} }
@@ -147,13 +159,15 @@ public class CommonUtil {
public static String getAppVersion() { public static String getAppVersion() {
if (null == appVersion) { if (null == appVersion) {
Properties properties = new Properties(); Properties properties = new Properties();
try { try (var is = CommonUtil.class.getClassLoader().getResourceAsStream("app.properties")) {
properties.load(CommonUtil.class.getClassLoader().getResourceAsStream("app.properties")); if (is != null) {
properties.load(is);
if (!properties.isEmpty()) { if (!properties.isEmpty()) {
appVersion = properties.getProperty("app.version") + "build" + properties.getProperty("build"); appVersion = properties.getProperty("app.version") + "build" + properties.getProperty("build");
} }
}
} catch (IOException e) { } catch (IOException e) {
e.printStackTrace(); LOGGER.error("读取app.properties失败", e);
} }
} }
return appVersion; return appVersion;
@@ -10,6 +10,8 @@ import io.vertx.core.json.JsonObject;
import java.io.InputStream; import java.io.InputStream;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
/** /**
* 异步读取配置工具类 * 异步读取配置工具类
@@ -62,12 +64,35 @@ public class ConfigUtil {
// 异步获取配置 // 异步获取配置
// 成功直接完成 promise // 成功直接完成 promise
retriever.getConfig() retriever.getConfig()
.onSuccess(promise::complete) .onSuccess(config -> {
promise.complete(config);
retriever.close();
})
.onFailure(err -> { .onFailure(err -> {
// 配置读取失败,直接返回失败 Future retriever.close();
// 读取失败时,尝试从 resources/ 子目录读取(兼容 Docker 卷挂载场景)
String resourcesPath = "resources/" + path;
if (!path.startsWith("resources/") && Files.exists(Path.of(resourcesPath))) {
ConfigStoreOptions fallbackStore = new ConfigStoreOptions()
.setType("file")
.setFormat(format)
.setConfig(new JsonObject().put("path", resourcesPath));
ConfigRetriever fallbackRetriever = ConfigRetriever
.create(vertx, new ConfigRetrieverOptions().addStore(fallbackStore));
fallbackRetriever.getConfig()
.onSuccess(config -> {
promise.complete(config);
fallbackRetriever.close();
})
.onFailure(e2 -> {
promise.fail(new RuntimeException(
"读取配置文件失败: " + path + " (也尝试了 " + resourcesPath + ")", e2));
fallbackRetriever.close();
});
} else {
promise.fail(new RuntimeException( promise.fail(new RuntimeException(
"读取配置文件失败: " + path, err)); "读取配置文件失败: " + path, err));
retriever.close(); }
}); });
return promise.future(); return promise.future();
@@ -4,17 +4,41 @@ import io.vertx.core.Future;
import io.vertx.core.Promise; import io.vertx.core.Promise;
import java.util.concurrent.ExecutionException; import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
public class FutureUtils { public class FutureUtils {
/** 默认同步等待超时时间(秒) */
private static final long DEFAULT_TIMEOUT_SECONDS = 120;
public static <T> T getResult(Future<T> future) { public static <T> T getResult(Future<T> future) {
try { try {
return future.toCompletionStage().toCompletableFuture().get(); return future.toCompletionStage().toCompletableFuture()
} catch (InterruptedException | ExecutionException e) { .get(DEFAULT_TIMEOUT_SECONDS, TimeUnit.SECONDS);
throw new RuntimeException(e); } catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException("线程被中断", e);
} catch (TimeoutException e) {
throw new RuntimeException("等待Future超时(" + DEFAULT_TIMEOUT_SECONDS + "秒)", e);
} catch (ExecutionException e) {
Throwable cause = e.getCause();
throw new RuntimeException(cause != null ? cause : e);
} }
} }
public static <T> T getResult(Promise<T> promise) { public static <T> T getResult(Promise<T> promise) {
return promise.future().toCompletionStage().toCompletableFuture().join(); try {
return promise.future().toCompletionStage().toCompletableFuture()
.get(DEFAULT_TIMEOUT_SECONDS, TimeUnit.SECONDS);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException("线程被中断", e);
} catch (TimeoutException e) {
throw new RuntimeException("等待Promise超时(" + DEFAULT_TIMEOUT_SECONDS + "秒)", e);
} catch (ExecutionException e) {
Throwable cause = e.getCause();
throw new RuntimeException(cause != null ? cause : e);
}
} }
} }
@@ -1,7 +1,7 @@
package cn.qaiu.vx.core.util; package cn.qaiu.vx.core.util;
import java.util.HashMap;
import java.util.Map; import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
/** /**
* vertx 上下文外的本地容器 为不在vertx线程的方法传递数据 * vertx 上下文外的本地容器 为不在vertx线程的方法传递数据
@@ -10,11 +10,10 @@ import java.util.Map;
* @author <a href="https://qaiu.top">QAIU</a> * @author <a href="https://qaiu.top">QAIU</a>
*/ */
public class LocalConstant { public class LocalConstant {
private static final Map<String, Object> LOCAL_CONST = new HashMap<>(); private static final Map<String, Object> LOCAL_CONST = new ConcurrentHashMap<>();
public static Map<String, Object> put(String k, Object v) { public static Map<String, Object> put(String k, Object v) {
if (LOCAL_CONST.containsKey(k)) return LOCAL_CONST; LOCAL_CONST.putIfAbsent(k, v);
LOCAL_CONST.put(k, v);
return LOCAL_CONST; return LOCAL_CONST;
} }
@@ -36,16 +36,20 @@ public final class ParamUtil {
public static MultiMap paramsToMap(String paramString) { public static MultiMap paramsToMap(String paramString) {
MultiMap entries = MultiMap.caseInsensitiveMultiMap(); MultiMap entries = MultiMap.caseInsensitiveMultiMap();
if (paramString == null) return entries; if (paramString == null || paramString.isEmpty()) return entries;
String[] params = paramString.split("&"); String[] params = paramString.split("&");
if (params.length == 0) return entries; if (params.length == 0) return entries;
for (String param : params) { for (String param : params) {
String[] kv = param.split("="); if (param == null || param.isEmpty()) {
continue;
}
String[] kv = param.split("=", 2);
if (kv.length == 2) { if (kv.length == 2) {
entries.set(kv[0], kv[1]); entries.set(kv[0], kv[1]);
} else { } else if (kv.length == 1) {
entries.set(kv[0], ""); entries.set(kv[0], "");
} }
// kv.length == 0 时(空字符串),跳过
} }
return entries; return entries;
} }
@@ -24,6 +24,10 @@ import java.lang.reflect.Method;
import java.net.URL; import java.net.URL;
import java.text.ParseException; import java.text.ParseException;
import java.util.*; import java.util.*;
import java.util.regex.Pattern;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import static cn.qaiu.vx.core.util.ConfigConstant.BASE_LOCATIONS; import static cn.qaiu.vx.core.util.ConfigConstant.BASE_LOCATIONS;
@@ -36,9 +40,17 @@ import static cn.qaiu.vx.core.util.ConfigConstant.BASE_LOCATIONS;
*/ */
public final class ReflectionUtil { public final class ReflectionUtil {
private static final Logger LOGGER = LoggerFactory.getLogger(ReflectionUtil.class);
// 缓存Reflections实例,避免重复扫描(每次扫描约35K+值,耗时1-3秒,占用大量内存) // 缓存Reflections实例,避免重复扫描(每次扫描约35K+值,耗时1-3秒,占用大量内存)
private static final Map<String, Reflections> REFLECTIONS_CACHE = new java.util.concurrent.ConcurrentHashMap<>(); private static final Map<String, Reflections> REFLECTIONS_CACHE = new java.util.concurrent.ConcurrentHashMap<>();
// 预编译的类型匹配正则,避免每次请求重新编译
private static final Pattern BASIC_TYPE_PATTERN = Pattern.compile(
"^java\\.lang\\.((Boolean)|(Character)|(Byte)|(Short)|(Integer)|(Long)|(Float)|(Double)|(String))$");
private static final Pattern BASIC_TYPE_ARRAY_PATTERN = Pattern.compile(
"^(boolean|char|byte|short|int|long|float|double|String)\\[]$");
/** /**
* 以默认配置的基础包路径获取反射器 * 以默认配置的基础包路径获取反射器
* *
@@ -128,7 +140,7 @@ public final class ReflectionUtil {
parameterTypes[j - k])); parameterTypes[j - k]));
} }
} catch (NotFoundException e) { } catch (NotFoundException e) {
e.printStackTrace(); LOGGER.error("获取方法参数失败", e);
} }
return paramMap; return paramMap;
} }
@@ -183,7 +195,7 @@ public final class ReflectionUtil {
try { try {
return DateUtils.parseDate(value, fmt); return DateUtils.parseDate(value, fmt);
} catch (ParseException e) { } catch (ParseException e) {
e.printStackTrace(); LOGGER.error("日期解析失败: {}", value, e);
throw new RuntimeException("无法将格式化日期"); throw new RuntimeException("无法将格式化日期");
} }
default: default:
@@ -215,7 +227,7 @@ public final class ReflectionUtil {
} }
return arr; return arr;
} catch (Exception e) { } catch (Exception e) {
e.printStackTrace(); LOGGER.error("数组类型转换失败: {}", value, e);
} }
return null; return null;
} }
@@ -229,8 +241,7 @@ public final class ReflectionUtil {
if (ctClass.isPrimitive() || "java.util.Date".equals(ctClass.getName())) { if (ctClass.isPrimitive() || "java.util.Date".equals(ctClass.getName())) {
return true; return true;
} }
return ctClass.getName().matches("^java\\.lang\\.((Boolean)|(Character)|(Byte)|(Short)|(Integer)|(Long)|" + return BASIC_TYPE_PATTERN.matcher(ctClass.getName()).matches();
"(Float)|(Double)|(String))$");
} }
/** /**
@@ -241,7 +252,7 @@ public final class ReflectionUtil {
public static boolean isBasicTypeArray(CtClass ctClass) { public static boolean isBasicTypeArray(CtClass ctClass) {
if (!ctClass.isArray()) { if (!ctClass.isArray()) {
return false; return false;
} else return (ctClass.getName().matches("^(boolen|char|byte|short|int|long|float|double|String)\\[]$")); } else return BASIC_TYPE_ARRAY_PATTERN.matcher(ctClass.getName()).matches();
} }
/** /**
@@ -12,14 +12,21 @@ import static io.vertx.core.http.HttpHeaders.CONTENT_TYPE;
public class ResponseUtil { public class ResponseUtil {
public static void redirect(HttpServerResponse response, String url) { public static void redirect(HttpServerResponse response, String url) {
if (response.ended() || response.closed()) {
return;
}
response.putHeader(CONTENT_TYPE, "text/html; charset=utf-8") response.putHeader(CONTENT_TYPE, "text/html; charset=utf-8")
.putHeader("Referrer-Policy", "no-referrer") .putHeader("Referrer-Policy", "no-referrer")
.putHeader(HttpHeaders.LOCATION, url).setStatusCode(302).end(); .putHeader(HttpHeaders.LOCATION, url).setStatusCode(302).end();
} }
public static void redirect(HttpServerResponse response, String url, Promise<?> promise) { public static void redirect(HttpServerResponse response, String url, Promise<?> promise) {
try {
redirect(response, url); redirect(response, url);
promise.complete(); promise.tryComplete();
} catch (Throwable t) {
promise.tryFail(t);
}
} }
public static void fireJsonObjectResponse(RoutingContext ctx, JsonObject jsonObject) { public static void fireJsonObjectResponse(RoutingContext ctx, JsonObject jsonObject) {
@@ -31,12 +38,18 @@ public class ResponseUtil {
} }
public static void fireJsonObjectResponse(RoutingContext ctx, JsonObject jsonObject, int statusCode) { public static void fireJsonObjectResponse(RoutingContext ctx, JsonObject jsonObject, int statusCode) {
if (ctx.response().ended() || ctx.response().closed()) {
return;
}
ctx.response().putHeader(CONTENT_TYPE, "application/json; charset=utf-8") ctx.response().putHeader(CONTENT_TYPE, "application/json; charset=utf-8")
.setStatusCode(statusCode) .setStatusCode(statusCode)
.end(jsonObject.encode()); .end(jsonObject.encode());
} }
public static void fireJsonObjectResponse(HttpServerResponse ctx, JsonObject jsonObject, int statusCode) { public static void fireJsonObjectResponse(HttpServerResponse ctx, JsonObject jsonObject, int statusCode) {
if (ctx.ended() || ctx.closed()) {
return;
}
ctx.putHeader(CONTENT_TYPE, "application/json; charset=utf-8") ctx.putHeader(CONTENT_TYPE, "application/json; charset=utf-8")
.setStatusCode(statusCode) .setStatusCode(statusCode)
.end(jsonObject.encode()); .end(jsonObject.encode());
@@ -55,10 +68,16 @@ public class ResponseUtil {
} }
public static void fireTextResponse(RoutingContext ctx, String text) { public static void fireTextResponse(RoutingContext ctx, String text) {
if (ctx.response().ended() || ctx.response().closed()) {
return;
}
ctx.response().putHeader(CONTENT_TYPE, "text/html; charset=utf-8").end(text); ctx.response().putHeader(CONTENT_TYPE, "text/html; charset=utf-8").end(text);
} }
public static void sendError(RoutingContext ctx, int statusCode) { public static void sendError(RoutingContext ctx, int statusCode) {
if (ctx.response().ended() || ctx.response().closed()) {
return;
}
ctx.response().setStatusCode(statusCode).end(); ctx.response().setStatusCode(statusCode).end();
} }
} }
@@ -3,7 +3,6 @@ package cn.qaiu.vx.core.util;
import io.vertx.core.json.JsonArray; import io.vertx.core.json.JsonArray;
import io.vertx.core.json.JsonObject; import io.vertx.core.json.JsonObject;
import io.vertx.core.shareddata.LocalMap; import io.vertx.core.shareddata.LocalMap;
import io.vertx.core.shareddata.SharedData;
/** /**
* vertx 共享数据 * vertx 共享数据
@@ -13,10 +12,8 @@ import io.vertx.core.shareddata.SharedData;
*/ */
public class SharedDataUtil { public class SharedDataUtil {
private static final SharedData sharedData = VertxHolder.getVertxInstance().sharedData(); public static io.vertx.core.shareddata.SharedData shareData() {
return VertxHolder.getVertxInstance().sharedData();
public static SharedData shareData() {
return sharedData;
} }
public static LocalMap<String, Object> getLocalMap(String key) { public static LocalMap<String, Object> getLocalMap(String key) {
@@ -24,7 +21,7 @@ public class SharedDataUtil {
} }
public static <T> LocalMap<String, T> getLocalMapWithCast(String key) { public static <T> LocalMap<String, T> getLocalMapWithCast(String key) {
return sharedData.getLocalMap(key); return shareData().getLocalMap(key);
} }
public static JsonObject getJsonConfig(String key) { public static JsonObject getJsonConfig(String key) {
@@ -1,10 +1,13 @@
package cn.qaiu.vx.core.verticle; package cn.qaiu.vx.core.verticle;
import io.vertx.core.AbstractVerticle; import io.vertx.core.AbstractVerticle;
import io.vertx.core.Future;
import io.vertx.core.Promise;
import io.vertx.core.http.*; import io.vertx.core.http.*;
import io.vertx.core.json.JsonObject; import io.vertx.core.json.JsonObject;
import io.vertx.core.net.NetClient; import io.vertx.core.net.NetClient;
import io.vertx.core.net.NetClientOptions; import io.vertx.core.net.NetClientOptions;
import io.vertx.core.net.NetSocket;
import io.vertx.core.net.ProxyOptions; import io.vertx.core.net.ProxyOptions;
import org.apache.commons.lang3.StringUtils; import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger; import org.slf4j.Logger;
@@ -24,13 +27,16 @@ public class HttpProxyVerticle extends AbstractVerticle {
private HttpClient httpClient; private HttpClient httpClient;
private NetClient netClient; private NetClient netClient;
private HttpServer httpServer;
private volatile boolean stopping = false;
private JsonObject proxyPreConf; private JsonObject proxyPreConf;
private JsonObject proxyServerConf; private JsonObject proxyServerConf;
@Override @Override
public void start() { public void start(io.vertx.core.Promise<Void> startPromise) {
stopping = false;
proxyServerConf = ((JsonObject)vertx.sharedData().getLocalMap(LOCAL).get(GLOBAL_CONFIG)).getJsonObject("proxy-server"); proxyServerConf = ((JsonObject)vertx.sharedData().getLocalMap(LOCAL).get(GLOBAL_CONFIG)).getJsonObject("proxy-server");
proxyPreConf = ((JsonObject)vertx.sharedData().getLocalMap(LOCAL).get(GLOBAL_CONFIG)).getJsonObject("proxy-pre"); proxyPreConf = ((JsonObject)vertx.sharedData().getLocalMap(LOCAL).get(GLOBAL_CONFIG)).getJsonObject("proxy-pre");
Integer serverPort = proxyServerConf.getInteger("port"); Integer serverPort = proxyServerConf.getInteger("port");
@@ -41,7 +47,12 @@ public class HttpProxyVerticle extends AbstractVerticle {
} }
// 初始化 HTTP 客户端,用于向目标服务器发送 HTTP 请求 // 初始化 HTTP 客户端,用于向目标服务器发送 HTTP 请求
HttpClientOptions httpClientOptions = new HttpClientOptions(); HttpClientOptions httpClientOptions = new HttpClientOptions()
.setMaxPoolSize(64)
.setMaxWaitQueueSize(256)
.setConnectTimeout(15000)
.setIdleTimeout(60)
.setKeepAlive(true);
if (proxyOptions != null) { if (proxyOptions != null) {
httpClientOptions.setProxyOptions(proxyOptions); httpClientOptions.setProxyOptions(proxyOptions);
} }
@@ -54,14 +65,14 @@ public class HttpProxyVerticle extends AbstractVerticle {
httpServerOptions.setClientAuth(ClientAuth.REQUIRED); httpServerOptions.setClientAuth(ClientAuth.REQUIRED);
} }
HttpServer server = vertx.createHttpServer(); httpServer = vertx.createHttpServer(httpServerOptions);
server.requestHandler(this::handleClientRequest); httpServer.requestHandler(this::handleClientRequest);
// 初始化 NetClient,用于在 CONNECT 请求中建立 TCP 连接隧道 // 初始化 NetClient,用于在 CONNECT 请求中建立 TCP 连接隧道
NetClientOptions netClientOptions = new NetClientOptions(); NetClientOptions netClientOptions = new NetClientOptions();
if (proxyOptions != null) { if (proxyOptions != null) {
httpClientOptions.setProxyOptions(proxyOptions); netClientOptions.setProxyOptions(proxyOptions);
} }
netClient = vertx.createNetClient(netClientOptions netClient = vertx.createNetClient(netClientOptions
@@ -69,16 +80,22 @@ public class HttpProxyVerticle extends AbstractVerticle {
.setTrustAll(true)); .setTrustAll(true));
// 启动 HTTP 代理服务器 // 启动 HTTP 代理服务器
server.listen(serverPort) httpServer.listen(serverPort)
.onSuccess(res-> LOGGER.info("HTTP Proxy server started on port {}", serverPort)) .onSuccess(res -> {
.onFailure(err-> LOGGER.error("Failed to start HTTP Proxy server: " + err.getMessage())); LOGGER.info("HTTP Proxy server started on port {}", serverPort);
startPromise.complete();
})
.onFailure(err -> {
LOGGER.error("Failed to start HTTP Proxy server: " + err.getMessage(), err);
closeClients().onComplete(close -> startPromise.fail(err));
});
} }
// 处理 HTTP CONNECT 请求,用于代理 HTTPS 流量 // 处理 HTTP CONNECT 请求,用于代理 HTTPS 流量
private void handleConnectRequest(HttpServerRequest clientRequest) { private void handleConnectRequest(HttpServerRequest clientRequest) {
String[] uriParts = clientRequest.uri().split(":"); String[] uriParts = clientRequest.uri().split(":");
if (uriParts.length != 2) { if (uriParts.length != 2) {
clientRequest.response().setStatusCode(400).end("Bad Request: Invalid URI format"); failClientResponse(clientRequest.response(), 400, "Bad Request: Invalid URI format");
return; return;
} }
@@ -88,19 +105,27 @@ public class HttpProxyVerticle extends AbstractVerticle {
try { try {
targetPort = Integer.parseInt(uriParts[1]); targetPort = Integer.parseInt(uriParts[1]);
} catch (NumberFormatException e) { } catch (NumberFormatException e) {
clientRequest.response().setStatusCode(400).end("Bad Request: Invalid port"); failClientResponse(clientRequest.response(), 400, "Bad Request: Invalid port");
return; return;
} }
clientRequest.pause(); clientRequest.pause();
// 通过 NetClient 连接目标服务器并创建隧道 // 通过 NetClient 连接目标服务器并创建隧道
try {
netClient.connect(targetPort, targetHost) netClient.connect(targetPort, targetHost)
.onSuccess(targetSocket -> { .onSuccess(targetSocket -> {
// Upgrade client connection to NetSocket and implement bidirectional data flow // Upgrade client connection to NetSocket and implement bidirectional data flow
clientRequest.toNetSocket() clientRequest.toNetSocket()
.onSuccess(clientSocket -> { .onSuccess(clientSocket -> {
// Set up bidirectional data forwarding clientSocket.pipeTo(targetSocket)
clientSocket.handler(targetSocket::write); .onFailure(err -> {
targetSocket.handler(clientSocket::write); LOGGER.debug("CONNECT client -> target pipe closed", err);
closeTunnelSockets(clientSocket, targetSocket);
});
targetSocket.pipeTo(clientSocket)
.onFailure(err -> {
LOGGER.debug("CONNECT target -> client pipe closed", err);
closeTunnelSockets(clientSocket, targetSocket);
});
// Close the other socket when one side closes // Close the other socket when one side closes
clientSocket.closeHandler(v -> targetSocket.close()); clientSocket.closeHandler(v -> targetSocket.close());
@@ -109,38 +134,55 @@ public class HttpProxyVerticle extends AbstractVerticle {
.onFailure(clientSocketAttempt -> { .onFailure(clientSocketAttempt -> {
System.err.println("Failed to upgrade client connection to socket: " + clientSocketAttempt.getMessage()); System.err.println("Failed to upgrade client connection to socket: " + clientSocketAttempt.getMessage());
targetSocket.close(); targetSocket.close();
clientRequest.response().setStatusCode(500).end("Internal Server Error"); failClientRequestAndClose(clientRequest, 500, "Internal Server Error");
}); });
}) })
.onFailure(connectionAttempt -> { .onFailure(connectionAttempt -> {
System.err.println("Failed to connect to target: " + connectionAttempt.getMessage()); LOGGER.warn("Failed to connect to target: {}", connectionAttempt.getMessage());
clientRequest.response().setStatusCode(502).end("Bad Gateway: Unable to connect to target"); failClientRequestAndClose(clientRequest, 502, "Bad Gateway: Unable to connect to target");
}); });
} catch (Exception e) {
LOGGER.warn("CONNECT 请求创建失败", e);
failClientRequestAndClose(clientRequest, 502, "Bad Gateway: Unable to connect to target");
}
} }
// 处理客户端的 HTTP 请求 // 处理客户端的 HTTP 请求
private void handleClientRequest(HttpServerRequest clientRequest) { private void handleClientRequest(HttpServerRequest clientRequest) {
if (stopping) {
failClientResponse(clientRequest.response(), 503, "Service Unavailable");
return;
}
// 打印来源ip和访问目标URI // 打印来源ip和访问目标URI
LOGGER.debug("source: {}, target: {}", clientRequest.remoteAddress().toString(), clientRequest.uri()); LOGGER.debug("source: {}, target: {}", clientRequest.remoteAddress().toString(), clientRequest.uri());
if (proxyServerConf.containsKey("username") && if (proxyServerConf.containsKey("username") &&
StringUtils.isNotBlank(proxyServerConf.getString("username"))) { StringUtils.isNotBlank(proxyServerConf.getString("username"))) {
String s = clientRequest.headers().get("Proxy-Authorization"); String s = clientRequest.headers().get("Proxy-Authorization");
if (s == null) { if (s == null) {
clientRequest.response().setStatusCode(403).end(); failClientResponse(clientRequest.response(), 403, null);
return;
}
String[] split;
try {
split = new String(Base64.getDecoder().decode(s.replace("Basic ", ""))).split(":");
} catch (IllegalArgumentException e) {
LOGGER.warn("Proxy-Authorization header is not valid Base64");
failClientResponse(clientRequest.response(), 403, null);
return;
}
if (split.length <= 1) {
LOGGER.warn("Proxy-Authorization header format invalid: missing username:password separator");
failClientResponse(clientRequest.response(), 403, null);
return; return;
} }
String[] split = new String(Base64.getDecoder().decode(s.replace("Basic ", ""))).split(":");
if (split.length > 1) {
// TODO
String username = proxyServerConf.getString("username"); String username = proxyServerConf.getString("username");
String password = proxyServerConf.getString("password"); String password = proxyServerConf.getString("password");
if (!split[0].equals(username) || !split[1].equals(password)) { if (!split[0].equals(username) || !split[1].equals(password)) {
LOGGER.info("-----auth failed------\nusername: {}\npassword: {}", username, password); LOGGER.info("-----auth failed------\nusername: {}", split[0]);
clientRequest.response().setStatusCode(403).end(); failClientResponse(clientRequest.response(), 403, null);
return; return;
} }
} }
}
if (clientRequest.method() == HttpMethod.CONNECT) { if (clientRequest.method() == HttpMethod.CONNECT) {
// 处理 CONNECT 请求 // 处理 CONNECT 请求
@@ -156,40 +198,147 @@ public class HttpProxyVerticle extends AbstractVerticle {
// 获取目标主机 // 获取目标主机
String hostHeader = clientRequest.getHeader("Host"); String hostHeader = clientRequest.getHeader("Host");
if (hostHeader == null) { if (hostHeader == null) {
clientRequest.response().setStatusCode(400).end("Host header is missing"); failClientResponse(clientRequest.response(), 400, "Host header is missing");
return; return;
} }
String targetHost = hostHeader.split(":")[0]; HostAndPort target;
int targetPort = extractPortFromUrl(clientRequest.uri()); // 默认为 HTTP 的端口 try {
clientRequest.pause(); // 暂停客户端请求的读取,避免数据丢失 target = parseHostHeader(hostHeader);
} catch (IllegalArgumentException e) {
failClientResponse(clientRequest.response(), 400, "Bad Request: Invalid Host header");
return;
}
String targetHost = target.host();
int targetPort = extractPortFromUrl(clientRequest.uri(), target.port()); // 默认为 HTTP 的端口
if (targetPort <= 0) {
failClientResponse(clientRequest.response(), 400, "Bad Request: Invalid target port");
return;
}
clientRequest.pause(); // 暂停客户端请求的读取,等上游请求创建完成
try {
httpClient.request(clientRequest.method(), targetPort, targetHost, clientRequest.uri()) httpClient.request(clientRequest.method(), targetPort, targetHost, clientRequest.uri())
.onSuccess(request -> { .onSuccess(request -> {
clientRequest.resume(); // 恢复客户端请求的读取
// 逐个设置请求头 // 逐个设置请求头
clientRequest.headers().forEach(header -> request.putHeader(header.getKey(), header.getValue())); clientRequest.headers().forEach(header -> request.putHeader(header.getKey(), header.getValue()));
// 将客户端请求的 body 转发给目标服务器 request.response()
clientRequest.bodyHandler(body ->
request.send(body)
.onSuccess(response -> { .onSuccess(response -> {
clientRequest.response().setStatusCode(response.statusCode()); HttpServerResponse clientResponse = clientRequest.response();
clientRequest.response().headers().setAll(response.headers()); if (clientResponse.ended() || clientResponse.closed()) {
response.body() response.resume();
.onSuccess(b -> clientRequest.response().end(b)) return;
.onFailure(err -> clientRequest.response() }
.setStatusCode(502).end("Bad Gateway: Unable to reach target")); clientResponse.setStatusCode(response.statusCode());
}) clientResponse.headers().setAll(response.headers());
.onFailure(err -> clientRequest.response() response.pipeTo(clientResponse)
.setStatusCode(502).end("Bad Gateway: Unable to reach target")) .onFailure(err -> {
); LOGGER.error("HTTP代理响应转发失败", err);
try {
response.request().reset();
} catch (Exception e) {
LOGGER.debug("HTTP代理上游响应已关闭", e);
}
failClientRequestAndClose(clientRequest, 502, "Bad Gateway: Unable to reach target");
});
}) })
.onFailure(err -> { .onFailure(err -> {
err.printStackTrace(); LOGGER.error("HTTP代理响应失败", err);
clientRequest.response().setStatusCode(502).end("Bad Gateway: Request failed"); try {
request.reset();
} catch (Exception e) {
LOGGER.debug("HTTP代理上游请求已关闭", e);
}
failClientRequestAndClose(clientRequest, 502, "Bad Gateway: Unable to reach target");
}); });
clientRequest.pipeTo(request)
.onFailure(err -> {
LOGGER.error("HTTP代理请求转发失败", err);
try {
request.reset();
} catch (Exception e) {
LOGGER.debug("HTTP代理上游请求已关闭", e);
}
failClientRequestAndClose(clientRequest, 502, "Bad Gateway: Unable to reach target");
});
clientRequest.resume();
})
.onFailure(err -> {
LOGGER.error("HTTP请求失败", err);
failClientRequestAndClose(clientRequest, 502, "Bad Gateway: Request failed");
});
} catch (Exception e) {
LOGGER.error("HTTP请求创建失败", e);
failClientRequestAndClose(clientRequest, 502, "Bad Gateway: Request failed");
}
}
private void failClientResponse(HttpServerResponse response, String message) {
failClientResponse(response, 502, message);
}
private void failClientResponse(HttpServerResponse response, int statusCode, String message) {
if (response.ended() || response.closed()) {
return;
}
try {
if (!response.headWritten()) {
response.setStatusCode(statusCode);
if (message == null) {
response.end();
} else {
response.end(message);
}
} else {
response.reset();
}
} catch (Exception e) {
LOGGER.debug("客户端响应已关闭,忽略代理错误响应", e);
}
}
private void failClientRequestAndClose(HttpServerRequest request, int statusCode, String message) {
HttpServerResponse response = request.response();
if (response.ended() || response.closed()) {
closeClientConnection(request);
return;
}
try {
if (!response.headWritten()) {
response.setStatusCode(statusCode);
Future<Void> endFuture = message == null ? response.end() : response.end(message);
endFuture.onComplete(v -> closeClientConnection(request));
} else {
response.reset();
closeClientConnection(request);
}
} catch (Exception e) {
LOGGER.debug("客户端响应已关闭,关闭代理连接", e);
closeClientConnection(request);
}
}
private void closeClientConnection(HttpServerRequest request) {
try {
request.connection().close();
} catch (Exception e) {
LOGGER.debug("关闭客户端代理连接失败", e);
}
}
private void closeTunnelSockets(NetSocket clientSocket, NetSocket targetSocket) {
try {
clientSocket.close();
} catch (Exception e) {
LOGGER.debug("关闭CONNECT客户端socket失败", e);
}
try {
targetSocket.close();
} catch (Exception e) {
LOGGER.debug("关闭CONNECT目标socket失败", e);
}
} }
@@ -200,6 +349,10 @@ public class HttpProxyVerticle extends AbstractVerticle {
* @return 提取的端口号,如果没有指定端口,则返回默认端口 * @return 提取的端口号,如果没有指定端口,则返回默认端口
*/ */
public static int extractPortFromUrl(String urlString) { public static int extractPortFromUrl(String urlString) {
return extractPortFromUrl(urlString, 80);
}
public static int extractPortFromUrl(String urlString, int defaultPort) {
try { try {
URI uri = new URI(urlString); URI uri = new URI(urlString);
int port = uri.getPort(); int port = uri.getPort();
@@ -208,27 +361,59 @@ public class HttpProxyVerticle extends AbstractVerticle {
if ("https".equalsIgnoreCase(uri.getScheme())) { if ("https".equalsIgnoreCase(uri.getScheme())) {
port = 443; // HTTPS 默认端口 port = 443; // HTTPS 默认端口
} else { } else {
port = 80; // HTTP 默认端口 port = defaultPort; // HTTP 默认端口
} }
} }
return port; return port;
} catch (Exception e) { } catch (Exception e) {
e.printStackTrace(); LOGGER.error("提取端口失败: {}", urlString, e);
// 出现异常时返回 -1,表示提取失败 // 出现异常时返回 -1,表示提取失败
return -1; return -1;
} }
} }
private HostAndPort parseHostHeader(String hostHeader) {
if (hostHeader.startsWith("[")) {
int end = hostHeader.indexOf(']');
if (end > 0) {
String host = hostHeader.substring(1, end);
int port = 80;
if (hostHeader.length() > end + 2 && hostHeader.charAt(end + 1) == ':') {
port = Integer.parseInt(hostHeader.substring(end + 2));
}
return new HostAndPort(host, port);
}
}
int lastColon = hostHeader.lastIndexOf(':');
if (lastColon > 0 && hostHeader.indexOf(':') == lastColon) {
return new HostAndPort(hostHeader.substring(0, lastColon), Integer.parseInt(hostHeader.substring(lastColon + 1)));
}
return new HostAndPort(hostHeader, 80);
}
private record HostAndPort(String host, int port) {
}
@Override @Override
public void stop() { public void stop(Promise<Void> stopPromise) {
// 停止 HTTP 客户端以释放资源 stopping = true;
if (httpClient != null) { Future<Void> serverClose = httpServer == null ? Future.succeededFuture() : httpServer.close();
httpClient.close(); serverClose.onComplete(serverResult -> closeClients().onComplete(clientResult -> {
if (serverResult.failed()) {
stopPromise.fail(serverResult.cause());
} else if (clientResult.failed()) {
stopPromise.fail(clientResult.cause());
} else {
stopPromise.complete();
} }
if (netClient != null) { }));
netClient.close();
} }
private Future<Void> closeClients() {
Future<Void> httpClientClose = httpClient == null ? Future.succeededFuture() : httpClient.close();
Future<Void> netClientClose = netClient == null ? Future.succeededFuture() : netClient.close();
return Future.all(httpClientClose, netClientClose).mapEmpty();
} }
} }
@@ -3,17 +3,20 @@ package cn.qaiu.vx.core.verticle;
import cn.qaiu.vx.core.util.*; import cn.qaiu.vx.core.util.*;
import io.vertx.core.AbstractVerticle; import io.vertx.core.AbstractVerticle;
import io.vertx.core.Future; import io.vertx.core.Future;
import io.vertx.core.Handler;
import io.vertx.core.Promise; import io.vertx.core.Promise;
import io.vertx.core.http.HttpClient; import io.vertx.core.http.HttpClient;
import io.vertx.core.http.HttpClientOptions; import io.vertx.core.http.HttpClientOptions;
import io.vertx.core.http.HttpServer; import io.vertx.core.http.HttpServer;
import io.vertx.core.http.HttpServerOptions; import io.vertx.core.http.HttpServerOptions;
import io.vertx.core.http.HttpServerRequest; import io.vertx.core.http.HttpServerRequest;
import io.vertx.core.http.HttpServerResponse;
import io.vertx.core.json.JsonArray; import io.vertx.core.json.JsonArray;
import io.vertx.core.json.JsonObject; import io.vertx.core.json.JsonObject;
import io.vertx.core.net.PemKeyCertOptions; import io.vertx.core.net.PemKeyCertOptions;
import io.vertx.ext.web.Route; import io.vertx.ext.web.Route;
import io.vertx.ext.web.Router; import io.vertx.ext.web.Router;
import io.vertx.ext.web.RoutingContext;
import io.vertx.ext.web.handler.StaticHandler; import io.vertx.ext.web.handler.StaticHandler;
import io.vertx.ext.web.proxy.handler.ProxyHandler; import io.vertx.ext.web.proxy.handler.ProxyHandler;
import io.vertx.httpproxy.HttpProxy; import io.vertx.httpproxy.HttpProxy;
@@ -27,7 +30,9 @@ import org.slf4j.LoggerFactory;
import java.io.File; import java.io.File;
import java.net.MalformedURLException; import java.net.MalformedURLException;
import java.net.URL; import java.net.URL;
import java.util.ArrayList;
import java.util.HashSet; import java.util.HashSet;
import java.util.List;
import java.util.Map; import java.util.Map;
import java.util.Set; import java.util.Set;
import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentHashMap;
@@ -45,10 +50,6 @@ public class ReverseProxyVerticle extends AbstractVerticle {
private static final Logger LOGGER = LoggerFactory.getLogger(ReverseProxyVerticle.class); private static final Logger LOGGER = LoggerFactory.getLogger(ReverseProxyVerticle.class);
private static final String PATH_PROXY_CONFIG = SharedDataUtil
.getJsonConfig(ConfigConstant.GLOBAL_CONFIG)
.getString("proxyConf");
private static final Future<JsonObject> CONFIG = ConfigUtil.readYamlConfig(PATH_PROXY_CONFIG);
private static final String DEFAULT_PATH_404 = "webroot/err/page404.html"; private static final String DEFAULT_PATH_404 = "webroot/err/page404.html";
private static String serverName = "Vert.x-proxy-server"; //Server name in Http response header private static String serverName = "Vert.x-proxy-server"; //Server name in Http response header
@@ -58,26 +59,40 @@ public class ReverseProxyVerticle extends AbstractVerticle {
/** /**
* 【优化】HttpClient连接池,按host:port缓存复用,避免每个请求都创建新连接 * 【优化】HttpClient连接池,按host:port缓存复用,避免每个请求都创建新连接
*/ */
private final Map<String, HttpClient> httpClientPool = new ConcurrentHashMap<>(); private final Map<String, HttpClientEntry> httpClientPool = new ConcurrentHashMap<>();
private final List<HttpServer> httpServers = new ArrayList<>();
private volatile boolean stopping = false;
/**
* 连接池条目。HttpProxy 会持有这里的 HttpClient 引用,不能在路由仍可用时关闭。
*/
private static class HttpClientEntry {
final HttpClient client;
HttpClientEntry(HttpClient client) {
this.client = client;
}
}
/** /**
* 【优化】高并发场景下的HttpClient配置 * 【优化】高并发场景下的HttpClient配置
*/ */
private static final int MAX_POOL_SIZE = 100; // 最大连接池大小 private static final int MAX_POOL_SIZE = 32; // 最大连接池大小
private static final int MAX_WAIT_QUEUE_SIZE = 500; // 最大等待队列大小 private static final int MAX_WAIT_QUEUE_SIZE = 128; // 最大等待队列大小
private static final int CONNECT_TIMEOUT = 30000; // 连接超时30秒 private static final int CONNECT_TIMEOUT = 30000; // 连接超时30秒
private static final int IDLE_TIMEOUT = 60; // 空闲超时60秒 private static final int IDLE_TIMEOUT = 60; // 空闲超时60秒
private static final boolean KEEP_ALIVE = true; // 启用Keep-Alive private static final boolean KEEP_ALIVE = true; // 启用Keep-Alive
private static final boolean PIPELINING = true; // 启用HTTP管线化 private static final boolean PIPELINING = false; // 代理场景关闭管线化,避免慢响应堆积
@Override @Override
public void start(Promise<Void> startPromise) { public void start(Promise<Void> startPromise) {
CONFIG.onSuccess(this::handleProxyConfList).onFailure(e -> { stopping = false;
String pathProxyConfig = SharedDataUtil
.getJsonConfig(ConfigConstant.GLOBAL_CONFIG)
.getString("proxyConf");
ConfigUtil.readYamlConfig(pathProxyConfig).onSuccess(config -> startProxyServers(config).onComplete(startPromise)).onFailure(e -> {
LOGGER.info("web代理配置已禁用,当前仅支持API调用"); LOGGER.info("web代理配置已禁用,当前仅支持API调用");
});
// createFileListener
startPromise.complete(); startPromise.complete();
});
} }
/** /**
@@ -85,16 +100,49 @@ public class ReverseProxyVerticle extends AbstractVerticle {
*/ */
@Override @Override
public void stop(Promise<Void> stopPromise) { public void stop(Promise<Void> stopPromise) {
LOGGER.info("Stopping ReverseProxyVerticle, closing {} HttpClient connections...", httpClientPool.size()); stopping = true;
httpClientPool.values().forEach(client -> { LOGGER.info("Stopping ReverseProxyVerticle, closing {} servers and {} HttpClient connections...",
httpServers.size(), httpClientPool.size());
List<Future<Void>> serverCloseFutures = new ArrayList<>();
httpServers.forEach(server -> serverCloseFutures.add(server.close()));
Future<Void> serverCloseFuture = serverCloseFutures.isEmpty()
? Future.succeededFuture()
: Future.all(serverCloseFutures).mapEmpty();
serverCloseFuture.onComplete(serverClose -> {
List<Future<Void>> clientCloseFutures = new ArrayList<>();
closeHttpClients(clientCloseFutures);
Future<Void> clientCloseFuture = clientCloseFutures.isEmpty()
? Future.succeededFuture()
: Future.all(clientCloseFutures).mapEmpty();
clientCloseFuture.onComplete(clientClose -> {
if (serverClose.succeeded()) {
httpServers.clear();
}
if (clientClose.succeeded()) {
httpClientPool.clear();
}
if (serverClose.failed()) {
stopPromise.fail(serverClose.cause());
} else if (clientClose.failed()) {
stopPromise.fail(clientClose.cause());
} else {
stopPromise.complete();
}
});
});
}
private void closeHttpClients(List<Future<Void>> closeFutures) {
httpClientPool.values().forEach(entry -> {
try { try {
client.close(); closeFutures.add(entry.client.close());
} catch (Exception e) { } catch (Exception e) {
LOGGER.warn("Error closing HttpClient: {}", e.getMessage()); LOGGER.warn("Error closing HttpClient: {}", e.getMessage());
} }
}); });
httpClientPool.clear();
stopPromise.complete();
} }
/** /**
@@ -105,7 +153,7 @@ public class ReverseProxyVerticle extends AbstractVerticle {
*/ */
private HttpClient getOrCreateHttpClient(String host, int port) { private HttpClient getOrCreateHttpClient(String host, int port) {
String key = host + ":" + port; String key = host + ":" + port;
return httpClientPool.computeIfAbsent(key, k -> { HttpClientEntry entry = httpClientPool.computeIfAbsent(key, k -> {
LOGGER.info("Creating new HttpClient for {}", key); LOGGER.info("Creating new HttpClient for {}", key);
HttpClientOptions options = new HttpClientOptions() HttpClientOptions options = new HttpClientOptions()
.setMaxPoolSize(MAX_POOL_SIZE) // 连接池大小 .setMaxPoolSize(MAX_POOL_SIZE) // 连接池大小
@@ -116,15 +164,16 @@ public class ReverseProxyVerticle extends AbstractVerticle {
.setKeepAliveTimeout(120) // Keep-Alive超时120秒 .setKeepAliveTimeout(120) // Keep-Alive超时120秒
.setPipelining(PIPELINING) // HTTP管线化 .setPipelining(PIPELINING) // HTTP管线化
.setPipeliningLimit(10) // 管线化限制 .setPipeliningLimit(10) // 管线化限制
.setDecompressionSupported(true) // 支持解压响应 .setDecompressionSupported(false) // 代理不解压,避免放大内存
.setTcpKeepAlive(true) // TCP Keep-Alive .setTcpKeepAlive(true) // TCP Keep-Alive
.setTcpNoDelay(true) // 禁用Nagle算法,降低延迟 .setTcpNoDelay(true) // 禁用Nagle算法,降低延迟
.setTcpFastOpen(true) // 启用TCP Fast Open .setTcpFastOpen(true) // 启用TCP Fast Open
.setTcpQuickAck(true) // 启用TCP Quick ACK .setTcpQuickAck(true) // 启用TCP Quick ACK
.setReuseAddress(true) // 允许地址重用 .setReuseAddress(true) // 允许地址重用
.setReusePort(true); // 允许端口重用 .setReusePort(true); // 允许端口重用
return vertx.createHttpClient(options); return new HttpClientEntry(vertx.createHttpClient(options));
}); });
return entry.client;
} }
/** /**
@@ -137,7 +186,7 @@ public class ReverseProxyVerticle extends AbstractVerticle {
* *
* @param config proxy config * @param config proxy config
*/ */
private void handleProxyConfList(JsonObject config) { private Future<Void> startProxyServers(JsonObject config) {
serverName = config.getString("server-name"); serverName = config.getString("server-name");
// 解析全局 trusted-proxies // 解析全局 trusted-proxies
JsonArray trustedArr = config.getJsonArray("trusted-proxies"); JsonArray trustedArr = config.getJsonArray("trusted-proxies");
@@ -149,13 +198,15 @@ public class ReverseProxyVerticle extends AbstractVerticle {
}); });
} }
JsonArray proxyConfList = config.getJsonArray("proxy"); JsonArray proxyConfList = config.getJsonArray("proxy");
List<Future<Void>> listenFutures = new ArrayList<>();
if (proxyConfList != null) { if (proxyConfList != null) {
proxyConfList.forEach(proxyConf -> { proxyConfList.forEach(proxyConf -> {
if (proxyConf instanceof JsonObject) { if (proxyConf instanceof JsonObject) {
handleProxyConf((JsonObject) proxyConf); listenFutures.add(handleProxyConf((JsonObject) proxyConf));
} }
}); });
} }
return listenFutures.isEmpty() ? Future.succeededFuture() : Future.all(listenFutures).mapEmpty();
} }
/** /**
@@ -201,7 +252,7 @@ public class ReverseProxyVerticle extends AbstractVerticle {
* *
* @param proxyConf 代理配置 * @param proxyConf 代理配置
*/ */
private void handleProxyConf(JsonObject proxyConf) { private Future<Void> handleProxyConf(JsonObject proxyConf) {
// page404 path // page404 path
if (proxyConf.containsKey( if (proxyConf.containsKey(
@@ -226,6 +277,10 @@ public class ReverseProxyVerticle extends AbstractVerticle {
// Add Server name header // Add Server name header
proxyRouter.route().handler(ctx -> { proxyRouter.route().handler(ctx -> {
if (stopping) {
sendProxyError(ctx, 503, "Service Unavailable");
return;
}
String realPath = ctx.request().uri(); String realPath = ctx.request().uri();
if (realPath.startsWith(REROUTE_PATH_PREFIX)) { if (realPath.startsWith(REROUTE_PATH_PREFIX)) {
// vertx web proxy暂不支持rewrite, 所以这里进行手动替换, 请求地址中的请求path前缀替换为originPath // vertx web proxy暂不支持rewrite, 所以这里进行手动替换, 请求地址中的请求path前缀替换为originPath
@@ -234,7 +289,9 @@ public class ReverseProxyVerticle extends AbstractVerticle {
return; return;
} }
if (!ctx.response().ended() && !ctx.response().closed()) {
ctx.response().putHeader("Server", serverName); ctx.response().putHeader("Server", serverName);
}
ctx.next(); ctx.next();
}); });
@@ -250,15 +307,19 @@ public class ReverseProxyVerticle extends AbstractVerticle {
// Send page404 page // Send page404 page
proxyRouter.errorHandler(404, ctx -> { proxyRouter.errorHandler(404, ctx -> {
ctx.response().sendFile(proxyConf.getString("page404")); sendNotFoundPage(ctx, proxyConf.getString("page404"));
}); });
proxyRouter.errorHandler(500, this::handleProxyFailure);
HttpServer server = getHttpsServer(proxyConf); HttpServer server = getHttpsServer(proxyConf);
server.requestHandler(proxyRouter); server.requestHandler(proxyRouter);
Integer port = proxyConf.getInteger("listen"); Integer port = proxyConf.getInteger("listen");
LOGGER.info("proxy server start on {} port", port); LOGGER.info("proxy server start on {} port", port);
server.listen(port); return server.listen(port)
.onSuccess(s -> httpServers.add(s))
.onFailure(e -> LOGGER.error("proxy server start failed on {} port", port, e))
.mapEmpty();
} }
private HttpServer getHttpsServer(JsonObject proxyConf) { private HttpServer getHttpsServer(JsonObject proxyConf) {
@@ -267,7 +328,7 @@ public class ReverseProxyVerticle extends AbstractVerticle {
.setTcpKeepAlive(true) // TCP Keep-Alive .setTcpKeepAlive(true) // TCP Keep-Alive
.setTcpNoDelay(true) // 禁用Nagle算法 .setTcpNoDelay(true) // 禁用Nagle算法
.setCompressionSupported(true) // 启用压缩 .setCompressionSupported(true) // 启用压缩
.setAcceptBacklog(50000) // 增加积压队列到50000 .setAcceptBacklog(1024) // 限制积压队列,避免小容器内存膨胀
.setIdleTimeout(120) // 空闲超时120秒 .setIdleTimeout(120) // 空闲超时120秒
.setTcpFastOpen(true) // 启用TCP Fast Open .setTcpFastOpen(true) // 启用TCP Fast Open
.setTcpQuickAck(true) // 启用TCP Quick ACK .setTcpQuickAck(true) // 启用TCP Quick ACK
@@ -303,6 +364,67 @@ public class ReverseProxyVerticle extends AbstractVerticle {
return vertx.createHttpServer(httpServerOptions); return vertx.createHttpServer(httpServerOptions);
} }
private void addProxyHandler(Route route, HttpProxy httpProxy) {
Handler<RoutingContext> proxyHandler = ProxyHandler.create(httpProxy);
route.handler(ctx -> {
try {
proxyHandler.handle(ctx);
} catch (Throwable t) {
LOGGER.error("反向代理处理异常", t);
ctx.fail(t);
}
}).failureHandler(this::handleProxyFailure);
}
private void handleProxyFailure(RoutingContext ctx) {
Throwable failure = ctx.failure();
if (failure != null) {
LOGGER.error("反向代理路由失败", failure);
}
int statusCode = ctx.statusCode() > 0 ? ctx.statusCode() : 502;
if (statusCode < 400) {
statusCode = 502;
}
sendProxyError(ctx, statusCode, "Bad Gateway");
}
private void sendNotFoundPage(RoutingContext ctx, String page404) {
HttpServerResponse response = ctx.response();
if (response.ended() || response.closed()) {
return;
}
try {
if (response.headWritten()) {
response.reset();
return;
}
response.sendFile(page404)
.onFailure(e -> {
LOGGER.warn("发送代理 404 页面失败: {}", page404, e);
sendProxyError(ctx, 404, "404 not found");
});
} catch (Exception e) {
LOGGER.warn("发送代理 404 页面异常: {}", page404, e);
sendProxyError(ctx, 404, "404 not found");
}
}
private void sendProxyError(RoutingContext ctx, int statusCode, String message) {
HttpServerResponse response = ctx.response();
if (response.ended() || response.closed()) {
return;
}
try {
if (!response.headWritten()) {
response.setStatusCode(statusCode).end(message);
} else {
response.reset();
}
} catch (Exception e) {
LOGGER.debug("代理响应已关闭,忽略错误响应", e);
}
}
/** /**
* 处理静态资源配置 * 处理静态资源配置
* *
@@ -351,7 +473,7 @@ public class ReverseProxyVerticle extends AbstractVerticle {
String host = url.getHost(); String host = url.getHost();
int port = url.getPort(); int port = url.getPort();
if (port == -1) { if (port == -1) {
port = 80; port = 443;
} }
String originPath = url.getPath(); String originPath = url.getPath();
LOGGER.info("path {}, originPath {}, to {}:{}", path, originPath, host, port); LOGGER.info("path {}, originPath {}, to {}:{}", path, originPath, host, port);
@@ -391,15 +513,14 @@ public class ReverseProxyVerticle extends AbstractVerticle {
if (StringUtils.isEmpty(originPath) || path.equals(originPath)) { if (StringUtils.isEmpty(originPath) || path.equals(originPath)) {
Route route = path.startsWith("~") ? proxyRouter.routeWithRegex(path.substring(1)) Route route = path.startsWith("~") ? proxyRouter.routeWithRegex(path.substring(1))
: proxyRouter.route(path); : proxyRouter.route(path);
// 【优化】为代理处理器添加超时 addProxyHandler(route, httpProxy);
route.handler(ProxyHandler.create(httpProxy));
} else { } else {
// 配置 /api/, / => 请求 /api/test 代理后 /test // 配置 /api/, / => 请求 /api/test 代理后 /test
// 配置 /api/, /xxx => 请求 /api/test 代理后 /xxx/test // 配置 /api/, /xxx => 请求 /api/test 代理后 /xxx/test
final String path0 = path; final String path0 = path;
final String originPath0 = REROUTE_PATH_PREFIX + originPath; final String originPath0 = REROUTE_PATH_PREFIX + originPath;
proxyRouter.route(originPath0 + "*").handler(ProxyHandler.create(httpProxy)); addProxyHandler(proxyRouter.route(originPath0 + "*"), httpProxy);
proxyRouter.route(path0 + "*").handler(ctx -> { proxyRouter.route(path0 + "*").handler(ctx -> {
String realPath = ctx.request().uri(); String realPath = ctx.request().uri();
if (realPath.startsWith(path0)) { if (realPath.startsWith(path0)) {
@@ -22,22 +22,22 @@ public class RouterVerticle extends AbstractVerticle {
private static final Logger LOGGER = LoggerFactory.getLogger(RouterVerticle.class); private static final Logger LOGGER = LoggerFactory.getLogger(RouterVerticle.class);
private static final int port = SharedDataUtil.getValueForServerConfig("port");
private static final Router router = new RouterHandlerFactory(
SharedDataUtil.getJsonStringForServerConfig("contextPath")).createRouter();
private static final JsonObject globalConfig = SharedDataUtil.getJsonConfig("globalConfig");
private HttpServer server; private HttpServer server;
private Router router;
private int port;
private JsonObject globalConfig;
static { static {
LOGGER.info(JacksonConfig.class.getSimpleName() + " >> "); LOGGER.info(JacksonConfig.class.getSimpleName() + " >> ");
JacksonConfig.nothing(); JacksonConfig.nothing();
LOGGER.info("To start listening to port {} ......", port);
} }
@Override @Override
public void start(Promise<Void> startPromise) { public void start(Promise<Void> startPromise) {
port = SharedDataUtil.getValueForServerConfig("port");
globalConfig = SharedDataUtil.getJsonConfig("globalConfig");
LOGGER.info("To start listening to port {} ......", port);
// 端口是否占用 // 端口是否占用
if (CommonUtil.isPortUsing(port)) { if (CommonUtil.isPortUsing(port)) {
throw new RuntimeException("Start fail: the '" + port + "' port is already in use..."); throw new RuntimeException("Start fail: the '" + port + "' port is already in use...");
@@ -61,9 +61,11 @@ public class RouterVerticle extends AbstractVerticle {
.setReuseAddress(true) // 允许地址重用 .setReuseAddress(true) // 允许地址重用
.setReusePort(true); // 允许端口重用 .setReusePort(true); // 允许端口重用
router = new RouterHandlerFactory(
SharedDataUtil.getJsonStringForServerConfig("contextPath")).createRouter();
server = vertx.createHttpServer(options); server = vertx.createHttpServer(options);
server.requestHandler(router).webSocketHandler(s->{}).listen() server.requestHandler(router).listen()
.onSuccess(s -> startPromise.complete()) .onSuccess(s -> startPromise.complete())
.onFailure(e -> startPromise.fail(e.getCause())); .onFailure(e -> startPromise.fail(e.getCause()));
} }
@@ -5,11 +5,15 @@ import cn.qaiu.vx.core.base.BaseAsyncService;
import cn.qaiu.vx.core.util.ReflectionUtil; import cn.qaiu.vx.core.util.ReflectionUtil;
import io.vertx.core.AbstractVerticle; import io.vertx.core.AbstractVerticle;
import io.vertx.core.Promise; import io.vertx.core.Promise;
import io.vertx.core.eventbus.MessageConsumer;
import io.vertx.core.json.JsonObject;
import io.vertx.serviceproxy.ServiceBinder; import io.vertx.serviceproxy.ServiceBinder;
import org.reflections.Reflections; import org.reflections.Reflections;
import org.slf4j.Logger; import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
import java.util.ArrayList;
import java.util.List;
import java.util.Set; import java.util.Set;
import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicInteger;
@@ -24,6 +28,7 @@ public class ServiceVerticle extends AbstractVerticle {
Logger LOGGER = LoggerFactory.getLogger(ServiceVerticle.class); Logger LOGGER = LoggerFactory.getLogger(ServiceVerticle.class);
private static final AtomicInteger ID = new AtomicInteger(1); private static final AtomicInteger ID = new AtomicInteger(1);
private static final Set<Class<?>> handlers; private static final Set<Class<?>> handlers;
private final List<MessageConsumer<JsonObject>> consumers = new ArrayList<>();
static { static {
Reflections reflections = ReflectionUtil.getReflections(); Reflections reflections = ReflectionUtil.getReflections();
@@ -39,7 +44,10 @@ public class ServiceVerticle extends AbstractVerticle {
try { try {
serviceNames.append(asyncService.getName()).append("|"); serviceNames.append(asyncService.getName()).append("|");
BaseAsyncService asInstance = (BaseAsyncService) ReflectionUtil.newWithNoParam(asyncService); BaseAsyncService asInstance = (BaseAsyncService) ReflectionUtil.newWithNoParam(asyncService);
binder.setAddress(asInstance.getAddress()).register(asInstance.getAsyncInterfaceClass(), asInstance); String address = asInstance.getAddress();
MessageConsumer<JsonObject> consumer = binder.setAddress(address)
.register(asInstance.getAsyncInterfaceClass(), asInstance);
consumers.add(consumer);
} catch (Exception e) { } catch (Exception e) {
LOGGER.error("Failed to register service: {}", asyncService.getName(), e); LOGGER.error("Failed to register service: {}", asyncService.getName(), e);
} }
@@ -49,4 +57,19 @@ public class ServiceVerticle extends AbstractVerticle {
} }
startPromise.complete(); startPromise.complete();
} }
@Override
public void stop(Promise<Void> stopPromise) {
int count = consumers.size();
consumers.forEach(consumer -> {
try {
consumer.unregister();
} catch (Exception e) {
LOGGER.warn("Failed to unregister service consumer at address: {}", consumer.address(), e);
}
});
consumers.clear();
LOGGER.info("ServiceVerticle stopped, unregistered {} services", count);
stopPromise.complete();
}
} }
@@ -32,7 +32,7 @@ public class HttpProxyConf {
public HttpProxyConf() { public HttpProxyConf() {
this.username = DEFAULT_USERNAME; this.username = DEFAULT_USERNAME;
this.password = DEFAULT_PASSWORD; this.password = DEFAULT_PASSWORD;
this.timeout = DEFAULT_PORT; this.port = DEFAULT_PORT;
this.timeout = DEFAULT_TIMEOUT; this.timeout = DEFAULT_TIMEOUT;
this.preProxyOptions = new ProxyOptions(); this.preProxyOptions = new ProxyOptions();
} }
+1 -1
View File
@@ -1,2 +1,2 @@
app.version=${project.version} app.version=${project.version}
build=${build.timestamp} build=${maven.build.timestamp}
+13
View File
@@ -0,0 +1,13 @@
#!/bin/sh
set -e
# Fix permissions on volume-mounted directories (runs as root)
chown -R appuser:appgroup /app/db /app/logs /app/resources 2>/dev/null || true
# Run Java directly - entrypoint is PID 1, exec makes Java PID 1
# Docker SIGTERM goes directly to Java, triggering ShutdownHook
DEFAULT_JVM_OPTS="-Xmx${JVM_XMX:-512M} -Xss${JVM_XSS:-512k} -XX:MaxDirectMemorySize=${JVM_MAX_DIRECT_MEMORY:-256M} -DNFD_LOG_LEVEL=${NFD_LOG_LEVEL:-info} -DNFD_PLAYGROUND_ENABLED=${NFD_PLAYGROUND_ENABLED:-false}"
if [ -n "${JVM_MAX_METASPACE:-}" ]; then
DEFAULT_JVM_OPTS="$DEFAULT_JVM_OPTS -XX:MaxMetaspaceSize=$JVM_MAX_METASPACE"
fi
exec java ${DEFAULT_JVM_OPTS} ${JVM_OPTS} -Duser.timezone=${TZ:-Asia/Shanghai} -jar /app/netdisk-fast-download.jar
+4 -4
View File
@@ -4,26 +4,26 @@ NFD 解析器模块:聚合各类网盘/分享页解析,统一输出文件列
- 语言:Java 17 - 语言:Java 17
- 构建:Maven - 构建:Maven
- 模块版本:10.1.17 - 模块版本:10.2.5
## 依赖(Maven Central ## 依赖(Maven Central
```xml ```xml
<dependency> <dependency>
<groupId>cn.qaiu</groupId> <groupId>cn.qaiu</groupId>
<artifactId>parser</artifactId> <artifactId>parser</artifactId>
<version>10.1.17</version> <version>10.2.5</version>
</dependency> </dependency>
``` ```
- Gradle Groovy DSL - Gradle Groovy DSL
```groovy ```groovy
dependencies { dependencies {
implementation 'cn.qaiu:parser:10.1.17' implementation 'cn.qaiu:parser:10.2.5'
} }
``` ```
- Gradle Kotlin DSL - Gradle Kotlin DSL
```kotlin ```kotlin
dependencies { dependencies {
implementation("cn.qaiu:parser:10.1.17") implementation("cn.qaiu:parser:10.2.5")
} }
``` ```
+1 -1
View File
@@ -28,7 +28,7 @@
<dependency> <dependency>
<groupId>cn.qaiu</groupId> <groupId>cn.qaiu</groupId>
<artifactId>parser</artifactId> <artifactId>parser</artifactId>
<version>10.1.17</version> <version>10.2.5</version>
</dependency> </dependency>
``` ```
+1 -1
View File
@@ -11,7 +11,7 @@
<dependency> <dependency>
<groupId>cn.qaiu</groupId> <groupId>cn.qaiu</groupId>
<artifactId>parser</artifactId> <artifactId>parser</artifactId>
<version>10.1.17</version> <version>10.2.5</version>
</dependency> </dependency>
``` ```
+3 -3
View File
@@ -240,13 +240,13 @@ var encoded = JsHttpClient.urlEncode("hello world"); // "hello%20world"
var decoded = JsHttpClient.urlDecode("hello%20world"); // "hello world" var decoded = JsHttpClient.urlDecode("hello%20world"); // "hello world"
// 发送简单表单数据 // 发送简单表单数据
var formResponse = http.sendForm({ var formResponse = http.sendForm("https://api.example.com/login", {
username: "user", username: "user",
password: "pass" password: "pass"
}); });
// 发送JSON数据 // 发送JSON数据
var jsonResponse = http.sendJson({ var jsonResponse = http.sendJson("https://api.example.com/submit", {
name: "test", name: "test",
value: 123 value: 123
}); });
@@ -637,7 +637,7 @@ A: 使用 `shareLinkInfo.getSharePassword()` 方法。
### Q: 如何处理需要登录的网盘? ### Q: 如何处理需要登录的网盘?
A: 使用 `http.putHeader()` 设置认证头,或使用 `http.sendForm()` 发送登录表单。 A: 使用 `http.putHeader()` 设置认证头,或使用 `http.sendForm(url, data)` 发送登录表单。
### Q: 如何解析复杂的HTML ### Q: 如何解析复杂的HTML
+1 -1
View File
@@ -68,7 +68,7 @@ List<FileInfo> files = tool.parseFileListSync();
``` ```
要点: 要点:
- 必须先 WebClientVertxInit.init(Vertx)若未显式初始化,内部将懒加载 Vertx.vertx(),建议显式注入以统一生命周期 - 必须先 WebClientVertxInit.init(Vertx)未初始化时会直接报错,避免解析器偷偷创建第二个 Vert.x 实例
- 支持三种同步方法: - 支持三种同步方法:
- `parseSync()`: 解析单个文件下载链接 - `parseSync()`: 解析单个文件下载链接
- `parseFileListSync()`: 解析文件列表 - `parseFileListSync()`: 解析文件列表
+1 -1
View File
@@ -17,7 +17,7 @@
this.temporaryExecutor = WebClientVertxInit.get().createSharedWorkerExecutor( this.temporaryExecutor = WebClientVertxInit.get().createSharedWorkerExecutor(
"playground-temp-" + System.currentTimeMillis(), "playground-temp-" + System.currentTimeMillis(),
1, // 每个请求只需要1个线程 1, // 每个请求只需要1个线程
10000000000L // 设置非常长的超时,避免被vertx强制中断 10000000000L // 设置非常长的超时,避免触发Vert.x阻塞线程告警
); );
// 执行完成或超时后关闭 // 执行完成或超时后关闭
+1 -1
View File
@@ -106,7 +106,7 @@ executionFuture.toCompletionStage()
### 长期方案(需大量工作) ### 长期方案(需大量工作)
1. **迁移到GraalVM JavaScript引擎** 1. **迁移到GraalVM JavaScript引擎**
- 支持CPU时间限制 - 支持CPU时间限制
- 可以强制中断 - 相比Nashorn更容易实现受控取消
- 更好的性能 - 更好的性能
- 但需要额外依赖 - 但需要额外依赖
+41 -7
View File
@@ -12,7 +12,7 @@
<groupId>cn.qaiu</groupId> <groupId>cn.qaiu</groupId>
<artifactId>parser</artifactId> <artifactId>parser</artifactId>
<version>10.2.5</version> <version>${parserVersion}</version>
<packaging>jar</packaging> <packaging>jar</packaging>
<name>cn.qaiu:parser</name> <name>cn.qaiu:parser</name>
@@ -35,9 +35,9 @@
</developers> </developers>
<scm> <scm>
<connection>scm:git:https://github.com/qaiu/netdisk-fast-download.git</connection> <connection>scm:git:https://github.com/${github.owner}/${github.repo}.git</connection>
<developerConnection>scm:git:ssh://[email protected]:qaiu/netdisk-fast-download.git</developerConnection> <developerConnection>scm:git:ssh://[email protected]:${github.owner}/${github.repo}.git</developerConnection>
<url>https://github.com/qaiu/netdisk-fast-download</url> <url>https://github.com/${github.owner}/${github.repo}</url>
</scm> </scm>
<distributionManagement> <distributionManagement>
@@ -52,20 +52,19 @@
</distributionManagement> </distributionManagement>
<properties> <properties>
<revision>0.2.1</revision>
<java.version>17</java.version> <java.version>17</java.version>
<maven.compiler.source>17</maven.compiler.source> <maven.compiler.source>17</maven.compiler.source>
<maven.compiler.target>17</maven.compiler.target> <maven.compiler.target>17</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<!-- Versions --> <!-- Versions -->
<vertx.version>4.5.24</vertx.version> <vertx.version>4.5.27</vertx.version>
<org.reflections.version>0.10.2</org.reflections.version> <org.reflections.version>0.10.2</org.reflections.version>
<lombok.version>1.18.38</lombok.version> <lombok.version>1.18.38</lombok.version>
<slf4j.version>2.0.16</slf4j.version> <slf4j.version>2.0.16</slf4j.version>
<commons-lang3.version>3.18.0</commons-lang3.version> <commons-lang3.version>3.18.0</commons-lang3.version>
<jackson.version>2.18.6</jackson.version> <jackson.version>2.18.6</jackson.version>
<logback.version>1.5.19</logback.version> <logback.version>1.5.32</logback.version>
<junit.version>4.13.2</junit.version> <junit.version>4.13.2</junit.version>
</properties> </properties>
@@ -124,6 +123,41 @@
<build> <build>
<plugins> <plugins>
<!-- 从 git remote origin 自动识别 GitHub 仓库地址 -->
<plugin>
<groupId>org.codehaus.gmavenplus</groupId>
<artifactId>gmavenplus-plugin</artifactId>
<version>4.1.1</version>
<dependencies>
<dependency>
<groupId>org.apache.groovy</groupId>
<artifactId>groovy</artifactId>
<version>4.0.24</version>
</dependency>
</dependencies>
<executions>
<execution>
<phase>initialize</phase>
<goals><goal>execute</goal></goals>
<configuration>
<scripts>
<script>
def url = 'git remote get-url origin'.execute().text.trim()
def m = (url =~ 'github\\.com[:/]([^/]+)/([^/.]+?)(?:\\.git)?$')
if (m.find()) {
project.properties.setProperty('github.owner', m.group(1))
project.properties.setProperty('github.repo', m.group(2))
} else {
project.properties.setProperty('github.owner', 'qaiu')
project.properties.setProperty('github.repo', 'netdisk-fast-download')
}
</script>
</scripts>
</configuration>
</execution>
</executions>
</plugin>
<!-- 编译 --> <!-- 编译 -->
<plugin> <plugin>
<groupId>org.apache.maven.plugins</groupId> <groupId>org.apache.maven.plugins</groupId>
@@ -7,12 +7,15 @@ import org.slf4j.LoggerFactory;
import cn.qaiu.parser.custom.CustomParserRegistry; import cn.qaiu.parser.custom.CustomParserRegistry;
public class WebClientVertxInit { public class WebClientVertxInit {
private Vertx vertx = null; private volatile Vertx vertx = null;
private static final WebClientVertxInit INSTANCE = new WebClientVertxInit(); private static final WebClientVertxInit INSTANCE = new WebClientVertxInit();
private static final Logger log = LoggerFactory.getLogger(WebClientVertxInit.class); private static final Logger log = LoggerFactory.getLogger(WebClientVertxInit.class);
public static void init(Vertx vx) { public static synchronized void init(Vertx vx) {
if (vx == null) {
throw new IllegalArgumentException("Vertx instance must not be null");
}
INSTANCE.vertx = vx; INSTANCE.vertx = vx;
// 自动加载JavaScript解析器脚本 // 自动加载JavaScript解析器脚本
@@ -23,17 +26,9 @@ public class WebClientVertxInit {
} }
} }
public static Vertx get() { public static synchronized Vertx get() {
if (INSTANCE.vertx == null) { if (INSTANCE.vertx == null) {
log.info("getVertx: Vertx实例不存在, 创建Vertx实例."); throw new IllegalStateException("Vertx实例未初始化,请先调用 WebClientVertxInit.init(vertx)");
INSTANCE.vertx = Vertx.vertx();
// 如果Vertx实例是新创建的,也尝试加载JavaScript脚本
try {
CustomParserRegistry.autoLoadJsScripts();
} catch (Exception e) {
log.warn("自动加载JavaScript解析器脚本失败", e);
}
} }
return INSTANCE.vertx; return INSTANCE.vertx;
} }
@@ -86,7 +86,10 @@ public class ShareLinkInfo {
// 将type和shareKey组合成一个字符串作为缓存key // 将type和shareKey组合成一个字符串作为缓存key
String key = type + ":" + shareKey; String key = type + ":" + shareKey;
if (type.equals("p115")) { if (type.equals("p115")) {
key += ("_" + otherParam.get("UA").toString().hashCode()); Object ua = otherParam != null ? otherParam.get("UA") : null;
if (ua != null) {
key += ("_" + ua.toString().hashCode());
}
} }
return key; return key;
} }
@@ -1,5 +1,6 @@
package cn.qaiu.parser;//package cn.qaiu.lz.common.parser; package cn.qaiu.parser;//package cn.qaiu.lz.common.parser;
import cn.qaiu.WebClientVertxInit;
import cn.qaiu.entity.FileInfo; import cn.qaiu.entity.FileInfo;
import cn.qaiu.entity.ShareLinkInfo; import cn.qaiu.entity.ShareLinkInfo;
import cn.qaiu.parser.clientlink.ClientLinkGeneratorFactory; import cn.qaiu.parser.clientlink.ClientLinkGeneratorFactory;
@@ -7,10 +8,26 @@ import cn.qaiu.parser.clientlink.ClientLinkType;
import io.vertx.core.Future; import io.vertx.core.Future;
import io.vertx.core.Promise; import io.vertx.core.Promise;
import java.util.function.Supplier;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.Map;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
public interface IPanTool { public interface IPanTool extends AutoCloseable {
/** 同步等待超时时间(秒) */
long SYNC_TIMEOUT_SECONDS = 120;
ScheduledExecutorService CLOSE_AFTER_SCHEDULER = Executors.newSingleThreadScheduledExecutor(r -> {
Thread t = new Thread(r, "pan-tool-close-after");
t.setDaemon(true);
return t;
});
/** /**
* 解析文件 * 解析文件
@@ -18,8 +35,72 @@ public interface IPanTool {
*/ */
Future<String> parse(); Future<String> parse();
static <T> Future<T> closeAfter(IPanTool tool, Supplier<Future<T>> action) {
Promise<T> promise = Promise.promise();
AtomicBoolean cleanupDone = new AtomicBoolean(false);
ScheduledFuture<?> cleanupTask = null;
try {
Future<T> future = action.get();
if (future == null) {
closeQuietly(tool);
return Future.failedFuture("解析器返回空 Future");
}
cleanupTask = CLOSE_AFTER_SCHEDULER.schedule(() -> {
if (cleanupDone.compareAndSet(false, true)) {
closeQuietly(tool);
failOnVertxContext(promise, "解析超时(" + SYNC_TIMEOUT_SECONDS + "秒)");
}
}, SYNC_TIMEOUT_SECONDS, TimeUnit.SECONDS);
ScheduledFuture<?> scheduledCleanupTask = cleanupTask;
future.onComplete(ar -> {
scheduledCleanupTask.cancel(false);
if (!cleanupDone.compareAndSet(false, true)) {
return;
}
closeQuietly(tool);
if (ar.succeeded()) {
promise.tryComplete(ar.result());
} else {
promise.tryFail(ar.cause());
}
});
return promise.future();
} catch (Throwable t) {
if (cleanupTask != null) {
cleanupTask.cancel(false);
}
closeQuietly(tool);
return Future.failedFuture(t);
}
}
private static <T> void failOnVertxContext(Promise<T> promise, String message) {
try {
WebClientVertxInit.get().runOnContext(ignored -> promise.tryFail(message));
} catch (Exception ignored) {
promise.tryFail(message);
}
}
static void closeQuietly(IPanTool tool) {
if (tool == null) {
return;
}
try {
tool.close();
} catch (Exception ignored) {
// ignore cleanup failures
}
}
static void shutdownCloseAfterScheduler() {
CLOSE_AFTER_SCHEDULER.shutdownNow();
}
default String parseSync() { default String parseSync() {
return parse().toCompletionStage().toCompletableFuture().join(); return timedJoin(parse());
} }
/** /**
@@ -33,7 +114,7 @@ public interface IPanTool {
} }
default List<FileInfo> parseFileListSync() { default List<FileInfo> parseFileListSync() {
return parseFileList().toCompletionStage().toCompletableFuture().join(); return timedJoin(parseFileList());
} }
/** /**
@@ -47,7 +128,7 @@ public interface IPanTool {
} }
default String parseByIdSync() { default String parseByIdSync() {
return parseById().toCompletionStage().toCompletableFuture().join(); return timedJoin(parseById());
} }
/** /**
@@ -126,7 +207,7 @@ public interface IPanTool {
* @return Map<ClientLinkType, String> 客户端下载链接集合 * @return Map<ClientLinkType, String> 客户端下载链接集合
*/ */
default Map<ClientLinkType, String> parseWithClientLinksSync() { default Map<ClientLinkType, String> parseWithClientLinksSync() {
return parseWithClientLinks().toCompletionStage().toCompletableFuture().join(); return timedJoin(parseWithClientLinks());
} }
/** /**
@@ -137,4 +218,26 @@ public interface IPanTool {
default ShareLinkInfo getShareLinkInfo() { default ShareLinkInfo getShareLinkInfo() {
return null; return null;
} }
@Override
default void close() {
// default no-op
}
/**
* 带超时的同步等待工具方法,替代无超时的 join()
*/
private static <T> T timedJoin(Future<T> future) {
try {
return future.toCompletionStage().toCompletableFuture()
.get(SYNC_TIMEOUT_SECONDS, TimeUnit.SECONDS);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException("线程被中断", e);
} catch (TimeoutException e) {
throw new RuntimeException("同步等待超时(" + SYNC_TIMEOUT_SECONDS + "秒)", e);
} catch (java.util.concurrent.ExecutionException e) {
throw new RuntimeException(e.getCause() != null ? e.getCause() : e);
}
}
} }
+307 -43
View File
@@ -20,6 +20,9 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
import java.io.*; import java.io.*;
import java.net.InetAddress;
import java.net.URL;
import java.net.UnknownHostException;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.util.Arrays; import java.util.Arrays;
import java.util.HashMap; import java.util.HashMap;
@@ -34,37 +37,66 @@ import java.util.zip.GZIPInputStream;
* <p>{网盘标识}Tool, 网盘标识不超过5个字符, 可以取网盘名称首字母缩写或拼音首字母, <br> * <p>{网盘标识}Tool, 网盘标识不超过5个字符, 可以取网盘名称首字母缩写或拼音首字母, <br>
* 音乐类型的解析以M开头, 例如网易云音乐Mne</p> * 音乐类型的解析以M开头, 例如网易云音乐Mne</p>
*/ */
public abstract class PanBase implements IPanTool { public abstract class PanBase implements IPanTool, Closeable {
protected Logger log = LoggerFactory.getLogger(this.getClass()); protected Logger log = LoggerFactory.getLogger(this.getClass());
protected Promise<String> promise = Promise.promise(); protected Promise<String> promise = Promise.promise();
/** private static final int MAX_COMPRESSED_RESPONSE_BYTES = 8 * 1024 * 1024;
* Http client private static final int MAX_DECOMPRESSED_RESPONSE_CHARS = 16 * 1024 * 1024;
*/ private static final int MAX_ERROR_BODY_CHARS = 4096;
protected WebClient client = WebClient.create(WebClientVertxInit.get(),
new WebClientOptions());
/** /**
* Http client session (会话管理, 带cookie请求) * 共享的 WebClient 配置(设置超时避免连接无限期占用)
*/
private static final WebClientOptions SHARED_OPTIONS = new WebClientOptions()
.setConnectTimeout(10000) // 连接超时 10 秒
.setIdleTimeout(30) // 空闲超时 30 秒
.setIdleTimeoutUnit(java.util.concurrent.TimeUnit.SECONDS);
private static final Object SHARED_CLIENT_LOCK = new Object();
/**
* 共享的 WebClient 实例(线程安全,避免每请求创建导致资源泄漏)
*/
private static volatile WebClient sharedClient;
private static volatile WebClient sharedClientNoRedirects;
private static volatile WebClient sharedClientDisableUA;
private static volatile boolean sharedClientsShutdown = false;
/**
* Http client (默认使用共享实例,代理模式下使用独立实例)
*/
protected WebClient client = sharedClient();
/**
* Http client session (会话管理, 带cookie请求, 每实例独立)
*/ */
protected WebClientSession clientSession = WebClientSession.create(client); protected WebClientSession clientSession = WebClientSession.create(client);
/** /**
* Http client 不自动跳转 * Http client 不自动跳转
*/ */
protected WebClient clientNoRedirects = WebClient.create(WebClientVertxInit.get(), protected WebClient clientNoRedirects = sharedClientNoRedirects();
new WebClientOptions().setFollowRedirects(false));
/** /**
* Http client disable UserAgent * Http client disable UserAgent
*/ */
protected WebClient clientDisableUA = WebClient.create(WebClientVertxInit.get() protected WebClient clientDisableUA = sharedClientDisableUA();
, new WebClientOptions().setUserAgentEnabled(false)
);
protected ShareLinkInfo shareLinkInfo; protected ShareLinkInfo shareLinkInfo;
/**
* 标记是否为代理模式(代理模式创建的 WebClient 需要手动关闭)
*/
private boolean isProxyMode = false;
/**
* 代理模式下创建的独立 WebClient 实例(需要在 close 时释放)
*/
private WebClient proxyClient = null;
private WebClient proxyClientNoRedirects = null;
/** /**
* 子类重写此构造方法不需要添加额外逻辑 * 子类重写此构造方法不需要添加额外逻辑
@@ -80,6 +112,7 @@ public abstract class PanBase implements IPanTool {
public PanBase(ShareLinkInfo shareLinkInfo) { public PanBase(ShareLinkInfo shareLinkInfo) {
this.shareLinkInfo = shareLinkInfo; this.shareLinkInfo = shareLinkInfo;
if (shareLinkInfo.getOtherParam().containsKey("proxy")) { if (shareLinkInfo.getOtherParam().containsKey("proxy")) {
this.isProxyMode = true;
JsonObject proxy = (JsonObject) shareLinkInfo.getOtherParam().get("proxy"); JsonObject proxy = (JsonObject) shareLinkInfo.getOtherParam().get("proxy");
ProxyOptions proxyOptions = new ProxyOptions() ProxyOptions proxyOptions = new ProxyOptions()
.setType(ProxyType.valueOf(proxy.getString("type").toUpperCase())) .setType(ProxyType.valueOf(proxy.getString("type").toUpperCase()))
@@ -91,22 +124,156 @@ public abstract class PanBase implements IPanTool {
if (StringUtils.isNotEmpty(proxy.getString("password"))) { if (StringUtils.isNotEmpty(proxy.getString("password"))) {
proxyOptions.setPassword(proxy.getString("password")); proxyOptions.setPassword(proxy.getString("password"));
} }
this.client = WebClient.create(WebClientVertxInit.get(), // 代理模式下创建独立的 WebClient 实例(应用超时配置)
new WebClientOptions() this.proxyClient = WebClient.create(WebClientVertxInit.get(),
new WebClientOptions(SHARED_OPTIONS)
.setUserAgentEnabled(false)
.setProxyOptions(proxyOptions));
this.proxyClientNoRedirects = WebClient.create(WebClientVertxInit.get(),
new WebClientOptions(SHARED_OPTIONS).setFollowRedirects(false)
.setUserAgentEnabled(false) .setUserAgentEnabled(false)
.setProxyOptions(proxyOptions)); .setProxyOptions(proxyOptions));
this.client = proxyClient;
this.clientSession = WebClientSession.create(client); this.clientSession = WebClientSession.create(client);
this.clientNoRedirects = WebClient.create(WebClientVertxInit.get(), this.clientNoRedirects = proxyClientNoRedirects;
new WebClientOptions().setFollowRedirects(false)
.setUserAgentEnabled(false)
.setProxyOptions(proxyOptions));
} }
} }
protected PanBase() { protected PanBase() {
} }
private static WebClient sharedClient() {
synchronized (SHARED_CLIENT_LOCK) {
if (sharedClientsShutdown) {
throw new IllegalStateException("共享 WebClient 已关闭");
}
if (sharedClient == null) {
sharedClient = WebClient.create(WebClientVertxInit.get(), new WebClientOptions(SHARED_OPTIONS));
}
return sharedClient;
}
}
private static WebClient sharedClientNoRedirects() {
synchronized (SHARED_CLIENT_LOCK) {
if (sharedClientsShutdown) {
throw new IllegalStateException("共享 WebClient 已关闭");
}
if (sharedClientNoRedirects == null) {
sharedClientNoRedirects = WebClient.create(WebClientVertxInit.get(),
new WebClientOptions(SHARED_OPTIONS).setFollowRedirects(false));
}
return sharedClientNoRedirects;
}
}
private static WebClient sharedClientDisableUA() {
synchronized (SHARED_CLIENT_LOCK) {
if (sharedClientsShutdown) {
throw new IllegalStateException("共享 WebClient 已关闭");
}
if (sharedClientDisableUA == null) {
sharedClientDisableUA = WebClient.create(WebClientVertxInit.get(),
new WebClientOptions(SHARED_OPTIONS).setUserAgentEnabled(false));
}
return sharedClientDisableUA;
}
}
public static void shutdownSharedClients() {
synchronized (SHARED_CLIENT_LOCK) {
sharedClientsShutdown = true;
closeSharedClient(sharedClient, "shared WebClient");
closeSharedClient(sharedClientNoRedirects, "shared WebClientNoRedirects");
closeSharedClient(sharedClientDisableUA, "shared WebClientDisableUA");
sharedClient = null;
sharedClientNoRedirects = null;
sharedClientDisableUA = null;
}
}
private static void closeSharedClient(WebClient client, String name) {
if (client == null) {
return;
}
try {
client.close();
} catch (Exception e) {
LoggerFactory.getLogger(PanBase.class).warn("关闭 {} 失败: {}", name, e.getMessage());
}
}
/**
* SSRF 防护: 校验通用自定义域名解析器 (CE/Ce4/Kd/Other 等) 即将请求的目标主机,
* 拒绝解析到回环/内网/链路本地/组播等非公网地址的域名, 阻止攻击者通过可控 DNS
* 记录 (或直接填写内网域名) 让服务端向内网/云元数据接口发起请求。
* <p>
* 必须在子类 parse() 中构造出 baseUrl/发起任何 clientSession 请求之前调用。
*
* @param url 从 shareLinkInfo.getShareUrl() 解析出的 URL
* @throws IOException 当主机无法解析或解析结果落入禁止的地址段时抛出
*/
protected static void assertPublicHost(URL url) throws IOException {
String host = url.getHost();
InetAddress[] addresses;
try {
addresses = InetAddress.getAllByName(host);
} catch (UnknownHostException e) {
throw new IOException("无法解析目标主机: " + host, e);
}
if (addresses.length == 0) {
throw new IOException("无法解析目标主机: " + host);
}
for (InetAddress addr : addresses) {
if (isDisallowedAddress(addr)) {
throw new IOException("目标地址不允许访问(内网/回环/链路本地/组播): "
+ host + " -> " + addr.getHostAddress());
}
}
}
/**
* 判断地址是否落入禁止访问的范围: 0.0.0.0/8, 10/8, 127/8, 169.254/16(含云元数据
* 169.254.169.254), 172.16/12, 192.168/16, 100.64/10(CGNAT), 224/4~255/4(组播/保留),
* 以及对应的 IPv6 回环/链路本地/唯一本地地址(fc00::/7)/组播地址; IPv4-映射的 IPv6
* 地址(::ffff:a.b.c.d) 会先还原为 IPv4 再判断,避免绕过。
*/
private static boolean isDisallowedAddress(InetAddress addr) {
byte[] bytes = addr.getAddress();
if (bytes.length == 16 && isIPv4Mapped(bytes)) {
byte[] v4 = new byte[4];
System.arraycopy(bytes, 12, v4, 0, 4);
bytes = v4;
}
if (bytes.length == 4) {
int b0 = bytes[0] & 0xFF;
int b1 = bytes[1] & 0xFF;
if (b0 == 0) return true; // 0.0.0.0/8
if (b0 == 10) return true; // 10.0.0.0/8
if (b0 == 127) return true; // 127.0.0.0/8 loopback
if (b0 == 169 && b1 == 254) return true; // 169.254.0.0/16 (含云元数据 169.254.169.254)
if (b0 == 172 && b1 >= 16 && b1 <= 31) return true; // 172.16.0.0/12
if (b0 == 192 && b1 == 168) return true; // 192.168.0.0/16
if (b0 == 100 && b1 >= 64 && b1 <= 127) return true;// 100.64.0.0/10 CGNAT
return b0 >= 224; // 224.0.0.0/4 组播 + 240.0.0.0/4 保留
}
// IPv6
if (addr.isAnyLocalAddress() || addr.isLoopbackAddress()
|| addr.isLinkLocalAddress() || addr.isSiteLocalAddress()
|| addr.isMulticastAddress()) {
return true;
}
return bytes.length == 16 && (bytes[0] & 0xFE) == 0xFC; // fc00::/7 unique local
}
private static boolean isIPv4Mapped(byte[] b) {
for (int i = 0; i < 10; i++) {
if (b[i] != 0) return false;
}
return (b[10] & 0xFF) == 0xFF && (b[11] & 0xFF) == 0xFF;
}
protected String baseMsg() { protected String baseMsg() {
if (shareLinkInfo.getShareUrl() != null) { if (shareLinkInfo.getShareUrl() != null) {
return shareLinkInfo.getPanName() + "-" + shareLinkInfo.getType() + ": url=" + shareLinkInfo.getShareUrl(); return shareLinkInfo.getPanName() + "-" + shareLinkInfo.getType() + ": url=" + shareLinkInfo.getShareUrl();
@@ -131,16 +298,19 @@ public abstract class PanBase implements IPanTool {
return; return;
} }
String s = String.format(errorMsg.replaceAll("\\{}", "%s"), args); String s = String.format(errorMsg.replaceAll("\\{}", "%s"), args);
log.error("解析异常: " + s, t.fillInStackTrace()); // 只记录异常消息和类型,不调用 fillInStackTrace 避免产生巨大栈信息
promise.fail(baseMsg() + ": 解析异常: " + s + " -> " + t); log.error("解析异常: {} - {}: {}", s, t.getClass().getSimpleName(), t.getMessage());
// 只传递异常消息,不传递完整异常对象,减少内存占用
String failMsg = baseMsg() + ": 解析异常: " + s + " -> " + t.getClass().getSimpleName() + ": " + t.getMessage();
promise.fail(failMsg);
} catch (Exception e) { } catch (Exception e) {
log.error("ErrorMsg format fail. The parameter has been discarded", e); log.error("ErrorMsg format fail. The parameter has been discarded", e);
log.error("解析异常: " + errorMsg, t.fillInStackTrace()); log.error("解析异常: {} - {}: {}", errorMsg, t.getClass().getSimpleName(), t.getMessage());
if (promise.future().isComplete()) { if (promise.future().isComplete()) {
log.warn("ErrorMsg format. Promise 已经完成, 无法再次失败: {}", errorMsg); log.warn("ErrorMsg format. Promise 已经完成, 无法再次失败: {}", errorMsg);
return; return;
} }
promise.fail(baseMsg() + ": 解析异常: " + errorMsg + " -> " + t); promise.fail(baseMsg() + ": 解析异常: " + errorMsg + " -> " + t.getClass().getSimpleName() + ": " + t.getMessage());
} }
} }
@@ -180,7 +350,7 @@ public abstract class PanBase implements IPanTool {
* @return Handler * @return Handler
*/ */
protected Handler<Throwable> handleFail(String errorMsg) { protected Handler<Throwable> handleFail(String errorMsg) {
return t -> fail(baseMsg() + " - 请求异常 {}: -> {}", errorMsg, t.fillInStackTrace()); return t -> fail(baseMsg() + " - 请求异常 {}: -> {}", errorMsg, t.getClass().getSimpleName() + ": " + t.getMessage());
} }
protected Handler<Throwable> handleFail() { protected Handler<Throwable> handleFail() {
@@ -199,28 +369,22 @@ public abstract class PanBase implements IPanTool {
String contentEncoding = res.getHeader("Content-Encoding"); String contentEncoding = res.getHeader("Content-Encoding");
try { try {
if ("gzip".equalsIgnoreCase(contentEncoding)) { if ("gzip".equalsIgnoreCase(contentEncoding)) {
// 如果是gzip压缩的响应体,解压 // 如果是gzip压缩的响应体,解压(只解压一次,缓存结果)
return new JsonObject(decompressGzip((Buffer) res.body())); String decompressed = decompressGzip((Buffer) res.body());
return new JsonObject(decompressed);
} else { } else {
return res.bodyAsJsonObject(); return res.bodyAsJsonObject();
} }
} catch (Exception e) { } catch (Exception e) {
if ("gzip".equalsIgnoreCase(contentEncoding)) { if ("gzip".equalsIgnoreCase(contentEncoding)) {
// 如果是gzip压缩的响应体,解压 // gzip解压失败,记录错误
try { log.error("响应gzip解压或JSON解析失败: {}", e.getMessage());
log.error(decompressGzip((Buffer) res.body())); fail("响应gzip解压或JSON解析失败: {}", e.getMessage());
fail(decompressGzip((Buffer) res.body()));
//throw new RuntimeException("响应不是JSON格式");
} catch (IOException ex) {
log.error("响应gzip解压失败");
fail("响应gzip解压失败: {}", ex.getMessage());
//throw new RuntimeException("响应gzip解压失败", ex);
}
} else { } else {
log.error("解析失败: json格式异常: {}", res.bodyAsString()); String bodyPreview = responseBodyPreview(res);
fail("解析失败: json格式异常: {}", res.bodyAsString()); log.error("解析失败: json格式异常: {}", bodyPreview);
//throw new RuntimeException("解析失败: json格式异常"); fail("解析失败: json格式异常: {}", bodyPreview);
} }
return JsonObject.of(); return JsonObject.of();
} }
@@ -250,7 +414,18 @@ public abstract class PanBase implements IPanTool {
protected void completeWithMeta(String url, Map<String, String> headers) { protected void completeWithMeta(String url, Map<String, String> headers) {
shareLinkInfo.getOtherParam().put("downloadUrl", url); shareLinkInfo.getOtherParam().put("downloadUrl", url);
if (headers != null && !headers.isEmpty()) { if (headers != null && !headers.isEmpty()) {
shareLinkInfo.getOtherParam().put("downloadHeaders", headers); // 过滤 null/空值,避免 cookie:null 覆盖入口参数或污染 curl 命令
Map<String, String> clean = new HashMap<>();
headers.forEach((k, v) -> {
if (k != null && v != null && !v.isBlank()) {
clean.put(k, v);
}
});
if (!clean.isEmpty()) {
shareLinkInfo.getOtherParam().put("downloadHeaders", clean);
// UC/夸克等需带 cookie 的直链,标记前端走下载器
shareLinkInfo.getOtherParam().put("needDownloader", true);
}
} }
promise.complete(url); promise.complete(url);
} }
@@ -283,11 +458,15 @@ public abstract class PanBase implements IPanTool {
if (iterator.hasNext()) { if (iterator.hasNext()) {
PanDomainTemplate next = iterator.next(); PanDomainTemplate next = iterator.next();
log.debug("规则不匹配, 处理解析器转发: {} -> {}", shareLinkInfo.getPanName(), next.getDisplayName()); log.debug("规则不匹配, 处理解析器转发: {} -> {}", shareLinkInfo.getPanName(), next.getDisplayName());
ParserCreate.fromType(next.name()) try {
IPanTool nextTool = ParserCreate.fromType(next.name())
.fromAnyShareUrl(shareLinkInfo.getShareUrl()) .fromAnyShareUrl(shareLinkInfo.getShareUrl())
.createTool() .createTool();
.parse() IPanTool.closeAfter(nextTool, nextTool::parse)
.onComplete(promise); .onComplete(promise);
} catch (Exception e) {
fail(e, "转发到下一个解析器失败: {}", next.getDisplayName());
}
} else { } else {
fail("error: 没有下一个解析处理器"); fail("error: 没有下一个解析处理器");
} }
@@ -303,6 +482,12 @@ public abstract class PanBase implements IPanTool {
* @throws IOException IOException * @throws IOException IOException
*/ */
private String decompressGzip(Buffer compressedData) throws IOException { private String decompressGzip(Buffer compressedData) throws IOException {
if (compressedData == null) {
return "";
}
if (compressedData.length() > MAX_COMPRESSED_RESPONSE_BYTES) {
throw new IOException("gzip响应体过大: " + compressedData.length() + " bytes");
}
try (ByteArrayInputStream bais = new ByteArrayInputStream(compressedData.getBytes()); try (ByteArrayInputStream bais = new ByteArrayInputStream(compressedData.getBytes());
GZIPInputStream gzis = new GZIPInputStream(bais); GZIPInputStream gzis = new GZIPInputStream(bais);
InputStreamReader isr = new InputStreamReader(gzis, StandardCharsets.UTF_8); InputStreamReader isr = new InputStreamReader(gzis, StandardCharsets.UTF_8);
@@ -311,18 +496,97 @@ public abstract class PanBase implements IPanTool {
char[] buffer = new char[4096]; char[] buffer = new char[4096];
int n; int n;
while ((n = isr.read(buffer)) != -1) { while ((n = isr.read(buffer)) != -1) {
writer.write(buffer, 0, n); writeLimited(writer, buffer, n);
} }
return writer.toString(); return writer.toString();
} }
} }
private void writeLimited(StringWriter writer, char[] buffer, int len) throws IOException {
if (writer.getBuffer().length() + len > MAX_DECOMPRESSED_RESPONSE_CHARS) {
throw new IOException("gzip解压后响应体过大");
}
writer.write(buffer, 0, len);
}
private String responseBodyPreview(HttpResponse<?> res) {
if (res == null || res.body() == null) {
return "";
}
try {
if (res.body() instanceof Buffer body) {
int length = Math.min(body.length(), MAX_ERROR_BODY_CHARS);
String preview = new String(body.getBytes(0, length), StandardCharsets.UTF_8);
return body.length() > length ? preview + "...(truncated " + body.length() + " bytes)" : preview;
}
String text = res.bodyAsString();
if (text == null || text.length() <= MAX_ERROR_BODY_CHARS) {
return text;
}
return text.substring(0, MAX_ERROR_BODY_CHARS) + "...(truncated " + text.length() + " chars)";
} catch (Exception e) {
return "<body preview failed: " + e.getMessage() + ">";
}
}
protected String getDomainName(){ protected String getDomainName(){
return shareLinkInfo.getOtherParam().getOrDefault("domainName", "").toString(); return shareLinkInfo.getOtherParam().getOrDefault("domainName", "").toString();
} }
/**
* 将入口请求中的加密 auth 透传到子目录/下载链接,避免进入子目录后丢失认证。
* otherParam 中的 key 为 {@code _authQuery}(由 web 层写入)。
*/
protected String appendAuthQuery(String url) {
if (StringUtils.isBlank(url) || shareLinkInfo == null || shareLinkInfo.getOtherParam() == null) {
return url;
}
Object authObj = shareLinkInfo.getOtherParam().get("_authQuery");
if (authObj == null) {
return url;
}
String auth = authObj.toString();
if (StringUtils.isBlank(auth)) {
return url;
}
// 已带 auth 则不再追加
if (url.contains("auth=")) {
return url;
}
try {
String encoded = java.net.URLEncoder.encode(auth, StandardCharsets.UTF_8);
return url + (url.contains("?") ? "&" : "?") + "auth=" + encoded;
} catch (Exception e) {
return url + (url.contains("?") ? "&" : "?") + "auth=" + auth;
}
}
@Override @Override
public ShareLinkInfo getShareLinkInfo() { public ShareLinkInfo getShareLinkInfo() {
return shareLinkInfo; return shareLinkInfo;
} }
/**
* 关闭代理模式下创建的 WebClient 资源
* 非代理模式使用共享实例,不需要关闭
*/
@Override
public void close() {
if (isProxyMode) {
try {
if (proxyClient != null) {
proxyClient.close();
}
} catch (Exception e) {
log.warn("关闭代理 WebClient 失败: {}", e.getMessage());
}
try {
if (proxyClientNoRedirects != null) {
proxyClientNoRedirects.close();
}
} catch (Exception e) {
log.warn("关闭代理 WebClientNoRedirects 失败: {}", e.getMessage());
}
}
}
} }
@@ -208,7 +208,7 @@ public enum PanDomainTemplate {
123795.com 123795.com
*/ */
YE("123网盘", YE("123网盘",
compile("https://www\\.(" + compile("https://(?:[a-zA-Z\\d-]+\\.)*(" +
"123254\\.com|" + "123254\\.com|" +
"123957\\.com|" + "123957\\.com|" +
"123295\\.com|" + "123295\\.com|" +
@@ -232,7 +232,7 @@ public enum PanDomainTemplate {
"123635\\.com|" + "123635\\.com|" +
"123242\\.com|" + "123242\\.com|" +
"123795\\.com" + "123795\\.com" +
")/s/(?<KEY>[a-zA-Z0-9_-]+)(?:\\.html)?"), ")/(?:(?:s|123pan)/|(?:[^/?#]+/)+)?(?<KEY>[a-zA-Z0-9]+-[a-zA-Z0-9]+|[a-zA-Z0-9_-]+)(?:\\.html)?(?:\\?.*)?"),
"https://www.123pan.com/s/{shareKey}", "https://www.123pan.com/s/{shareKey}",
Ye2Tool.class), Ye2Tool.class),
// https://www.ecpan.cn/web/#/yunpanProxy?path=%2F%23%2Fdrive%2Foutside&data={code}&isShare=1 // https://www.ecpan.cn/web/#/yunpanProxy?path=%2F%23%2Fdrive%2Foutside&data={code}&isShare=1
@@ -247,9 +247,15 @@ public enum PanDomainTemplate {
"https://cowtransfer.com/s/{shareKey}", "https://cowtransfer.com/s/{shareKey}",
CowTool.class), CowTool.class),
CT("城通网盘", CT("城通网盘",
compile("https://(?:[a-zA-Z\\d-]+\\.)?(ctfile|545c|u062|ghpym|474b)\\.com/f(ile)?/" + compile("https?://(?:[a-zA-Z\\d-]+\\.)?(ctfile|545c|u062|ghpym|474b)\\.com/f(ile)?/" +
"(?<KEY>[0-9a-zA-Z_-]+)(\\?p=(?<PWD>\\w+))?"), "(?<KEY>[0-9a-zA-Z_-]+)/?(?:\\?(?:(?:[^#&]*&)*p=(?<PWD>\\w+)(?:&[^#]*)?|[^#]*))?"),
"https://474b.com/file/{shareKey}", "https://ctfile.com/file/{shareKey}",
CtTool.class),
// https://url94.ctfile.com/d/64115194-164803691-48508c?p=7609&d=164803691&fk=decb36
CTD("城通网盘-目录",
compile("https?://(?:[a-zA-Z\\d-]+\\.)?(ctfile|545c|u062|ghpym|474b)\\.com/d/" +
"(?<KEY>[0-9a-zA-Z_-]+)/?(?:\\?(?:(?:[^#&]*&)*p=(?<PWD>\\w+)(?:&[^#]*)?|[^#]*))?"),
"https://ctfile.com/d/{shareKey}",
CtTool.class), CtTool.class),
// https://www.vyuyun.com/s/QMa6ie?password=I4KG7H // https://www.vyuyun.com/s/QMa6ie?password=I4KG7H
// https://www.vyuyun.com/s/QMa6ie/file?password=I4KG7H // https://www.vyuyun.com/s/QMa6ie/file?password=I4KG7H
@@ -371,6 +377,16 @@ public enum PanDomainTemplate {
MmgTool.class), MmgTool.class),
// =====================私有盘解析========================== // =====================私有盘解析==========================
// 永硕E盘空间分享:https://qaiu.ysepan.com/ (空间名即 shareKey,密码为空间访问密码)
// 主域名 ysepan.com / ys168.com;备用 cccpan.com / ysupan.com / uupan.net / ysok.net
YS("永硕E盘",
compile("https?://(?!(?:www|zy|ht|api|c\\d+|ys-[a-zA-Z0-9]+)\\.)(?<KEY>[a-zA-Z\\d-]+)\\."
+ "(?:ysepan\\.com|ys168\\.com|cccpan\\.com|ysupan\\.com|uupan\\.net|ysok\\.net)"
+ "/?(?:\\?.*)?"),
"https://{shareKey}.ysepan.com/",
"https://www.ysepan.com/",
YsTool.class),
// Cloudreve自定义域名解析, 解析器CeTool兜底策略, 即任意域名如果匹配不到对应的规则, 则由CeTool统一处理, // Cloudreve自定义域名解析, 解析器CeTool兜底策略, 即任意域名如果匹配不到对应的规则, 则由CeTool统一处理,
// 如果不属于Cloudreve盘 则调用下一个自定义域名解析器, 若都处理不了则抛出异常, 这种匹配模式类似责任链 // 如果不属于Cloudreve盘 则调用下一个自定义域名解析器, 若都处理不了则抛出异常, 这种匹配模式类似责任链
// http(s)://pan.huang1111.cn/s/xxx // http(s)://pan.huang1111.cn/s/xxx
@@ -9,6 +9,8 @@ import org.apache.commons.lang3.StringUtils;
import java.net.URLEncoder; import java.net.URLEncoder;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.util.EnumSet;
import java.util.Set;
import java.util.regex.Matcher; import java.util.regex.Matcher;
import static cn.qaiu.parser.PanDomainTemplate.KEY; import static cn.qaiu.parser.PanDomainTemplate.KEY;
@@ -23,6 +25,9 @@ import static cn.qaiu.parser.PanDomainTemplate.PWD;
* Create at 2024/9/15 14:10 * Create at 2024/9/15 14:10
*/ */
public class ParserCreate { public class ParserCreate {
private static final Set<PanDomainTemplate> GENERIC_BUILT_IN_PARSERS =
EnumSet.of(PanDomainTemplate.CE, PanDomainTemplate.KD, PanDomainTemplate.OTHER);
private final PanDomainTemplate panDomainTemplate; private final PanDomainTemplate panDomainTemplate;
private final ShareLinkInfo shareLinkInfo; private final ShareLinkInfo shareLinkInfo;
@@ -81,7 +86,7 @@ public class ParserCreate {
if (shareKey != null) { if (shareKey != null) {
shareLinkInfo.setShareKey(shareKey); shareLinkInfo.setShareKey(shareKey);
} }
} catch (Exception ignored) {} } catch (IllegalStateException | IllegalArgumentException ignored) {}
// 提取密码 // 提取密码
try { try {
@@ -89,7 +94,7 @@ public class ParserCreate {
if (StringUtils.isNotEmpty(pwd)) { if (StringUtils.isNotEmpty(pwd)) {
shareLinkInfo.setSharePassword(pwd); shareLinkInfo.setSharePassword(pwd);
} }
} catch (Exception ignored) {} } catch (IllegalStateException | IllegalArgumentException ignored) {}
// 设置标准URL // 设置标准URL
if (customParserConfig.getStandardUrlTemplate() != null) { if (customParserConfig.getStandardUrlTemplate() != null) {
@@ -132,12 +137,12 @@ public class ParserCreate {
if (StringUtils.isNotEmpty(pwd)) { if (StringUtils.isNotEmpty(pwd)) {
shareLinkInfo.setSharePassword(pwd); shareLinkInfo.setSharePassword(pwd);
} }
standardUrl = standardUrl.replace("{pwd}", pwd); standardUrl = standardUrl.replace("{pwd}", StringUtils.defaultString(pwd));
} catch (Exception ignored) {} } catch (IllegalStateException | IllegalArgumentException ignored) {}
shareLinkInfo.setShareUrl(shareUrl); shareLinkInfo.setShareUrl(shareUrl);
shareLinkInfo.setShareKey(shareKey); shareLinkInfo.setShareKey(shareKey);
if (!(panDomainTemplate.ordinal() >= PanDomainTemplate.CE.ordinal())) { if (!isGenericBuiltInParser(panDomainTemplate)) {
shareLinkInfo.setStandardUrl(standardUrl); shareLinkInfo.setStandardUrl(standardUrl);
} }
return this; return this;
@@ -197,7 +202,7 @@ public class ParserCreate {
} }
// 内置解析器处理 // 内置解析器处理
if (panDomainTemplate.ordinal() >= PanDomainTemplate.CE.ordinal()) { if (isGenericBuiltInParser(panDomainTemplate)) {
// 处理Cloudreve(ce)类: pan.huang1111.cn_s_wDz5TK _ -> / // 处理Cloudreve(ce)类: pan.huang1111.cn_s_wDz5TK _ -> /
String[] s = shareKey.split("_"); String[] s = shareKey.split("_");
String standardUrl = "https://" + String.join("/", s); String standardUrl = "https://" + String.join("/", s);
@@ -247,9 +252,19 @@ public class ParserCreate {
return this; return this;
} }
// 根据分享链接获取PanDomainTemplate实例(优先匹配自定义解析器) // 根据分享链接获取PanDomainTemplate实例
public synchronized static ParserCreate fromShareUrl(String shareUrl) { public synchronized static ParserCreate fromShareUrl(String shareUrl) {
// 优先查找支持正则匹配的自定义解析器 if (StringUtils.isBlank(shareUrl)) {
throw new IllegalArgumentException("shareUrl不能为空");
}
shareUrl = shareUrl.trim();
ParserCreate builtInParser = fromBuiltInShareUrl(shareUrl, false);
if (builtInParser != null) {
return builtInParser;
}
// 明确内置解析器未命中时,再查找支持正则匹配的自定义解析器
for (CustomParserConfig customConfig : CustomParserRegistry.getAll().values()) { for (CustomParserConfig customConfig : CustomParserRegistry.getAll().values()) {
if (customConfig.supportsFromShareUrl()) { if (customConfig.supportsFromShareUrl()) {
Matcher matcher = customConfig.getMatchPattern().matcher(shareUrl); Matcher matcher = customConfig.getMatchPattern().matcher(shareUrl);
@@ -266,14 +281,14 @@ public class ParserCreate {
if (shareKey != null) { if (shareKey != null) {
shareLinkInfo.setShareKey(shareKey); shareLinkInfo.setShareKey(shareKey);
} }
} catch (Exception ignored) {} } catch (IllegalStateException | IllegalArgumentException ignored) {}
try { try {
String password = matcher.group("PWD"); String password = matcher.group("PWD");
if (password != null) { if (password != null) {
shareLinkInfo.setSharePassword(password); shareLinkInfo.setSharePassword(password);
} }
} catch (Exception ignored) {} } catch (IllegalStateException | IllegalArgumentException ignored) {}
// 设置标准URL(如果有模板) // 设置标准URL(如果有模板)
if (customConfig.getStandardUrlTemplate() != null) { if (customConfig.getStandardUrlTemplate() != null) {
@@ -296,21 +311,34 @@ public class ParserCreate {
} }
} }
// 查找内置解析器 // 最后再走 Cloudreve/可道云/其他网盘这类泛化兜底,避免抢走自定义解析器
builtInParser = fromBuiltInShareUrl(shareUrl, true);
if (builtInParser != null) {
return builtInParser;
}
throw new IllegalArgumentException("Unsupported share URL");
}
private static ParserCreate fromBuiltInShareUrl(String shareUrl, boolean genericOnly) {
for (PanDomainTemplate panDomainTemplate : PanDomainTemplate.values()) { for (PanDomainTemplate panDomainTemplate : PanDomainTemplate.values()) {
boolean genericParser = isGenericBuiltInParser(panDomainTemplate);
if (genericOnly != genericParser) {
continue;
}
if (panDomainTemplate.getPattern().matcher(shareUrl).matches()) { if (panDomainTemplate.getPattern().matcher(shareUrl).matches()) {
ShareLinkInfo shareLinkInfo = ShareLinkInfo.newBuilder() ShareLinkInfo shareLinkInfo = ShareLinkInfo.newBuilder()
.type(panDomainTemplate.name().toLowerCase()) .type(panDomainTemplate.name().toLowerCase())
.panName(panDomainTemplate.getDisplayName()) .panName(panDomainTemplate.getDisplayName())
.shareUrl(shareUrl).build(); .shareUrl(shareUrl).build();
if (panDomainTemplate.ordinal() >= PanDomainTemplate.CE.ordinal()) { if (isGenericBuiltInParser(panDomainTemplate)) {
shareLinkInfo.setStandardUrl(shareUrl); shareLinkInfo.setStandardUrl(shareUrl);
} }
ParserCreate parserCreate = new ParserCreate(panDomainTemplate, shareLinkInfo); ParserCreate parserCreate = new ParserCreate(panDomainTemplate, shareLinkInfo);
return parserCreate.normalizeShareLink(); return parserCreate.normalizeShareLink();
} }
} }
throw new IllegalArgumentException("Unsupported share URL"); return null;
} }
// 根据type获取枚举实例(优先查找自定义解析器) // 根据type获取枚举实例(优先查找自定义解析器)
@@ -353,7 +381,7 @@ public class ParserCreate {
// 自定义解析器处理 // 自定义解析器处理
if (isCustomParser) { if (isCustomParser) {
path = this.shareLinkInfo.getType() + "/" + this.shareLinkInfo.getShareKey(); path = this.shareLinkInfo.getType() + "/" + this.shareLinkInfo.getShareKey();
} else if (panDomainTemplate.ordinal() >= PanDomainTemplate.CE.ordinal()) { } else if (isGenericBuiltInParser(panDomainTemplate)) {
// 处理Cloudreve(ce)类: pan.huang1111.cn_s_wDz5TK _ -> / // 处理Cloudreve(ce)类: pan.huang1111.cn_s_wDz5TK _ -> /
path = this.shareLinkInfo.getType() + "/" path = this.shareLinkInfo.getType() + "/"
+ this.shareLinkInfo.getShareUrl() + this.shareLinkInfo.getShareUrl()
@@ -382,6 +410,10 @@ public class ParserCreate {
return customParserConfig; return customParserConfig;
} }
private static boolean isGenericBuiltInParser(PanDomainTemplate panDomainTemplate) {
return GENERIC_BUILT_IN_PARSERS.contains(panDomainTemplate);
}
/** /**
* 获取内置解析器模板(仅当isCustomParser为false时有效) * 获取内置解析器模板(仅当isCustomParser为false时有效)
* @return 内置解析器模板,如果是自定义解析器则返回null * @return 内置解析器模板,如果是自定义解析器则返回null
@@ -0,0 +1,149 @@
package cn.qaiu.parser;
import java.util.concurrent.ConcurrentHashMap;
/**
* 解析器 Token/Cookie 缓存 — 支持多账号隔离。
* <p>
* 以 (diskType + "#" + accountKey) 作为缓存 key,不同账号的 token 互不覆盖。
* accountKey 优先使用 _configId,其次使用 username、cookie 前16位等可区分标识。
* </p>
*/
public final class TokenCache {
private TokenCache() {}
/** token 缓存 */
private static final ConcurrentHashMap<String, String> tokenMap = new ConcurrentHashMap<>();
/** 过期时间缓存(毫秒时间戳) */
private static final ConcurrentHashMap<String, Long> expireMap = new ConcurrentHashMap<>();
/** 同一 key 下的额外字符串缓存(如 userId) */
private static final ConcurrentHashMap<String, String> extraMap = new ConcurrentHashMap<>();
/** 布尔标记缓存(如 authFlag */
private static final ConcurrentHashMap<String, Boolean> flagMap = new ConcurrentHashMap<>();
// ============ key 构造 ============
public static String key(String diskType, String accountKey) {
return diskType + "#" + (accountKey == null ? "_default" : accountKey);
}
// ============ token ============
public static String getToken(String cacheKey) {
return tokenMap.get(cacheKey);
}
public static void putToken(String cacheKey, String token) {
if (token == null) {
tokenMap.remove(cacheKey);
} else {
tokenMap.put(cacheKey, token);
}
}
// ============ expire ============
public static long getExpire(String cacheKey) {
return expireMap.getOrDefault(cacheKey, 0L);
}
public static void putExpire(String cacheKey, long expireMs) {
expireMap.put(cacheKey, expireMs);
}
public static boolean isExpired(String cacheKey) {
long exp = getExpire(cacheKey);
return exp <= 0 || System.currentTimeMillis() > exp;
}
// ============ extra (userId 等) ============
public static String getExtra(String cacheKey) {
return extraMap.get(cacheKey);
}
public static void putExtra(String cacheKey, String value) {
if (value == null) {
extraMap.remove(cacheKey);
} else {
extraMap.put(cacheKey, value);
}
}
// ============ flag (authFlag 等) ============
public static boolean getFlag(String cacheKey, boolean defaultValue) {
return flagMap.getOrDefault(cacheKey, defaultValue);
}
public static void putFlag(String cacheKey, boolean value) {
flagMap.put(cacheKey, value);
}
// ============ 清除 ============
public static void remove(String cacheKey) {
tokenMap.remove(cacheKey);
expireMap.remove(cacheKey);
extraMap.remove(cacheKey);
flagMap.remove(cacheKey);
}
/**
* 清除指定网盘类型的所有缓存(精准清除,不影响其他网盘类型)
*/
public static void removeByDiskType(String diskType) {
String prefix = diskType + "#";
tokenMap.keySet().removeIf(k -> k.startsWith(prefix));
expireMap.keySet().removeIf(k -> k.startsWith(prefix));
extraMap.keySet().removeIf(k -> k.startsWith(prefix));
flagMap.keySet().removeIf(k -> k.startsWith(prefix));
}
public static void clear() {
tokenMap.clear();
expireMap.clear();
extraMap.clear();
flagMap.clear();
}
// ============ Token 持久化队列 ============
/** 待持久化的 cachedToken 数据 (cacheKey -> [token, expireMs]) */
private static final ConcurrentHashMap<String, String[]> persistQueue = new ConcurrentHashMap<>();
/** 待回写的凭据更新 (cacheKey -> newCredential),如 PaliTool refresh_token 轮换 */
private static final ConcurrentHashMap<String, String> credentialUpdateQueue = new ConcurrentHashMap<>();
/**
* 解析器登录成功后,将 token 加入持久化队列(下次 recordConfigUsage 回写 DB
*/
public static void queueCachedTokenPersist(String cacheKey, String token, long expireMs) {
if (cacheKey != null && token != null) {
persistQueue.put(cacheKey, new String[]{token, String.valueOf(expireMs)});
}
}
/**
* 凭据本身被替换(如 PaliTool refresh_token 轮换),加入回写队列
*/
public static void queueCredentialUpdate(String cacheKey, String newCredential) {
if (cacheKey != null && newCredential != null) {
credentialUpdateQueue.put(cacheKey, newCredential);
}
}
/**
* 消费持久化队列:返回 [token, expireMs] 并移除;无数据返回 null
*/
public static String[] pollCachedTokenPersist(String cacheKey) {
return cacheKey == null ? null : persistQueue.remove(cacheKey);
}
/**
* 消费凭据更新队列:返回新凭据并移除;无数据返回 null
*/
public static String pollCredentialUpdate(String cacheKey) {
return cacheKey == null ? null : credentialUpdateQueue.remove(cacheKey);
}
}
@@ -21,6 +21,7 @@ import java.util.concurrent.ConcurrentHashMap;
public class CustomParserRegistry { public class CustomParserRegistry {
private static final Logger log = LoggerFactory.getLogger(CustomParserRegistry.class); private static final Logger log = LoggerFactory.getLogger(CustomParserRegistry.class);
private static final int MAX_CUSTOM_PARSERS = Integer.getInteger("parser.custom.maxRegistrySize", 256);
/** /**
* 存储自定义解析器配置的Map,key为类型标识,value为配置对象 * 存储自定义解析器配置的Map,key为类型标识,value为配置对象
@@ -33,7 +34,7 @@ public class CustomParserRegistry {
* @param config 解析器配置 * @param config 解析器配置
* @throws IllegalArgumentException 如果type已存在或与内置解析器冲突 * @throws IllegalArgumentException 如果type已存在或与内置解析器冲突
*/ */
public static void register(CustomParserConfig config) { public static synchronized void register(CustomParserConfig config) {
if (config == null) { if (config == null) {
throw new IllegalArgumentException("config不能为空"); throw new IllegalArgumentException("config不能为空");
} }
@@ -59,6 +60,11 @@ public class CustomParserRegistry {
"类型标识 '" + type + "' 已被注册,请先注销或使用其他标识" "类型标识 '" + type + "' 已被注册,请先注销或使用其他标识"
); );
} }
if (CUSTOM_PARSERS.size() >= MAX_CUSTOM_PARSERS) {
throw new IllegalArgumentException(
"自定义解析器数量已达到上限(" + MAX_CUSTOM_PARSERS + "个),请先注销不需要的解析器"
);
}
CUSTOM_PARSERS.put(type, config); CUSTOM_PARSERS.put(type, config);
log.info("注册自定义解析器成功: {} ({})", config.getDisplayName(), type); log.info("注册自定义解析器成功: {} ({})", config.getDisplayName(), type);
@@ -171,7 +177,7 @@ public class CustomParserRegistry {
* @param type 解析器类型标识 * @param type 解析器类型标识
* @return 是否注销成功 * @return 是否注销成功
*/ */
public static boolean unregister(String type) { public static synchronized boolean unregister(String type) {
if (type == null || type.trim().isEmpty()) { if (type == null || type.trim().isEmpty()) {
return false; return false;
} }
@@ -213,7 +219,7 @@ public class CustomParserRegistry {
/** /**
* 清空所有自定义解析器 * 清空所有自定义解析器
*/ */
public static void clear() { public static synchronized void clear() {
CUSTOM_PARSERS.clear(); CUSTOM_PARSERS.clear();
} }
@@ -2,19 +2,21 @@ package cn.qaiu.parser.customjs;
import cn.qaiu.WebClientVertxInit; import cn.qaiu.WebClientVertxInit;
import cn.qaiu.util.HttpResponseHelper; import cn.qaiu.util.HttpResponseHelper;
import io.vertx.core.Future;
import io.vertx.core.MultiMap; import io.vertx.core.MultiMap;
import io.vertx.core.Promise; import io.vertx.core.Promise;
import io.vertx.core.buffer.Buffer; import io.vertx.core.buffer.Buffer;
import io.vertx.core.http.HttpClient;
import io.vertx.core.http.HttpClientOptions;
import io.vertx.core.http.HttpClientRequest;
import io.vertx.core.http.HttpClientResponse;
import io.vertx.core.http.HttpHeaders;
import io.vertx.core.http.HttpMethod;
import io.vertx.core.http.RequestOptions;
import io.vertx.core.json.Json;
import io.vertx.core.json.JsonObject; import io.vertx.core.json.JsonObject;
import io.vertx.core.net.ProxyOptions; import io.vertx.core.net.ProxyOptions;
import io.vertx.core.net.ProxyType; import io.vertx.core.net.ProxyType;
import io.vertx.ext.web.client.HttpRequest;
import io.vertx.ext.web.client.HttpResponse; import io.vertx.ext.web.client.HttpResponse;
import io.vertx.ext.web.client.WebClient;
import io.vertx.ext.web.client.WebClientOptions;
import io.vertx.ext.web.client.WebClientSession;
import io.vertx.ext.web.multipart.MultipartForm;
import org.apache.commons.lang3.StringUtils; import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger; import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
@@ -27,8 +29,13 @@ import java.net.UnknownHostException;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.util.HashMap; import java.util.HashMap;
import java.util.Map; import java.util.Map;
import java.util.Set;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException; import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import java.util.regex.Pattern; import java.util.regex.Pattern;
/** /**
@@ -41,9 +48,58 @@ import java.util.regex.Pattern;
public class JsHttpClient { public class JsHttpClient {
private static final Logger log = LoggerFactory.getLogger(JsHttpClient.class); private static final Logger log = LoggerFactory.getLogger(JsHttpClient.class);
private static final int MAX_RESPONSE_BODY_BYTES = 8 * 1024 * 1024;
private static final int MAX_REQUEST_BODY_BYTES = 8 * 1024 * 1024;
private static final int MAX_HEADER_COUNT = 64;
private static final int MAX_HEADER_VALUE_LENGTH = 4096;
private static final int MAX_TIMEOUT_SECONDS = 120;
private static final int MAX_REDIRECTS = 5;
private static final String DEFAULT_ACCEPT_ENCODING = "gzip, deflate, br";
private final WebClient client; private static final Object SHARED_CLIENT_LOCK = new Object();
private final WebClientSession clientSession; // 共享 HttpClient 实例(非代理模式),懒加载避免类初始化阶段抢跑 Vert.x。
private static volatile HttpClient sharedClient;
private static volatile boolean sharedClientShutdown = false;
/**
* 关闭共享 HttpClient(应用关闭时调用)
*/
public static void shutdownSharedClient() {
synchronized (SHARED_CLIENT_LOCK) {
sharedClientShutdown = true;
if (sharedClient != null) {
sharedClient.close();
sharedClient = null;
}
}
}
private static HttpClient sharedClient() {
synchronized (SHARED_CLIENT_LOCK) {
ensureSharedClientAvailable();
if (sharedClient == null) {
sharedClient = WebClientVertxInit.get().createHttpClient(
new HttpClientOptions()
.setConnectTimeout(10000)
.setIdleTimeout(30)
.setIdleTimeoutUnit(TimeUnit.SECONDS)
.setMaxPoolSize(64));
}
return sharedClient;
}
}
private static void ensureSharedClientAvailable() {
if (sharedClientShutdown) {
throw new IllegalStateException("共享 JavaScript HttpClient 已关闭");
}
}
private final HttpClient client;
private final boolean ownClient; // 标记是否为自建 client(需要 close
private final AtomicBoolean closed = new AtomicBoolean(false);
private final Object requestLock = new Object();
private final Set<HttpClientRequest> activeRequests = ConcurrentHashMap.newKeySet();
private MultiMap headers; private MultiMap headers;
private int timeoutSeconds = 30; // 默认超时时间30秒 private int timeoutSeconds = 30; // 默认超时时间30秒
@@ -61,11 +117,12 @@ public class JsHttpClient {
}; };
public JsHttpClient() { public JsHttpClient() {
this.client = WebClient.create(WebClientVertxInit.get(), new WebClientOptions());; ensureSharedClientAvailable();
this.clientSession = WebClientSession.create(client); this.client = sharedClient();
this.ownClient = false;
this.headers = MultiMap.caseInsensitiveMultiMap(); this.headers = MultiMap.caseInsensitiveMultiMap();
// 设置默认的Accept-Encoding头以支持压缩响应 // 设置默认的Accept-Encoding头以支持压缩响应
this.headers.set("Accept-Encoding", "gzip, deflate, br, zstd"); this.headers.set("Accept-Encoding", DEFAULT_ACCEPT_ENCODING);
// 设置默认的User-Agent头 // 设置默认的User-Agent头
this.headers.set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 Edg/140.0.0.0"); this.headers.set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 Edg/140.0.0.0");
// 设置默认的Accept-Language头 // 设置默认的Accept-Language头
@@ -77,6 +134,7 @@ public class JsHttpClient {
* @param proxyConfig 代理配置JsonObject,包含type、host、port、username、password * @param proxyConfig 代理配置JsonObject,包含type、host、port、username、password
*/ */
public JsHttpClient(JsonObject proxyConfig) { public JsHttpClient(JsonObject proxyConfig) {
ensureSharedClientAvailable();
if (proxyConfig != null && proxyConfig.containsKey("type")) { if (proxyConfig != null && proxyConfig.containsKey("type")) {
ProxyOptions proxyOptions = new ProxyOptions() ProxyOptions proxyOptions = new ProxyOptions()
.setType(ProxyType.valueOf(proxyConfig.getString("type").toUpperCase())) .setType(ProxyType.valueOf(proxyConfig.getString("type").toUpperCase()))
@@ -90,18 +148,21 @@ public class JsHttpClient {
proxyOptions.setPassword(proxyConfig.getString("password")); proxyOptions.setPassword(proxyConfig.getString("password"));
} }
this.client = WebClient.create(WebClientVertxInit.get(), this.client = WebClientVertxInit.get().createHttpClient(
new WebClientOptions() new HttpClientOptions()
.setUserAgentEnabled(false) .setConnectTimeout(10000)
.setIdleTimeout(30)
.setIdleTimeoutUnit(TimeUnit.SECONDS)
.setMaxPoolSize(16)
.setProxyOptions(proxyOptions)); .setProxyOptions(proxyOptions));
this.clientSession = WebClientSession.create(client); this.ownClient = true;
} else { } else {
this.client = WebClient.create(WebClientVertxInit.get()); this.client = sharedClient();
this.clientSession = WebClientSession.create(client); this.ownClient = false;
} }
this.headers = MultiMap.caseInsensitiveMultiMap(); this.headers = MultiMap.caseInsensitiveMultiMap();
// 设置默认的Accept-Encoding头以支持压缩响应 // 设置默认的Accept-Encoding头以支持压缩响应
this.headers.set("Accept-Encoding", "gzip, deflate, br, zstd"); this.headers.set("Accept-Encoding", DEFAULT_ACCEPT_ENCODING);
// 设置默认的User-Agent头 // 设置默认的User-Agent头
this.headers.set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 Edg/140.0.0.0"); this.headers.set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 Edg/140.0.0.0");
// 设置默认的Accept-Language头 // 设置默认的Accept-Language头
@@ -183,13 +244,7 @@ public class JsHttpClient {
*/ */
public JsHttpResponse get(String url) { public JsHttpResponse get(String url) {
validateUrlSecurity(url); validateUrlSecurity(url);
return executeRequest(() -> { return executeRequest(HttpMethod.GET, url, null, false);
HttpRequest<Buffer> request = client.getAbs(url);
if (!headers.isEmpty()) {
request.putHeaders(headers);
}
return request.send();
});
} }
/** /**
@@ -198,16 +253,25 @@ public class JsHttpClient {
* @return HTTP响应 * @return HTTP响应
*/ */
public JsHttpResponse getWithRedirect(String url) { public JsHttpResponse getWithRedirect(String url) {
validateUrlSecurity(url); String currentUrl = url;
return executeRequest(() -> { for (int redirectCount = 0; redirectCount <= MAX_REDIRECTS; redirectCount++) {
HttpRequest<Buffer> request = client.getAbs(url); validateUrlSecurity(currentUrl);
if (!headers.isEmpty()) { JsHttpResponse response = executeRequest(HttpMethod.GET, currentUrl, null, false);
request.putHeaders(headers); if (!isRedirectStatus(response.statusCode())) {
return response;
} }
// 设置跟随重定向
request.followRedirects(true); if (redirectCount == MAX_REDIRECTS) {
return request.send(); throw new RuntimeException("重定向次数超过限制: " + MAX_REDIRECTS);
}); }
String location = response.header(HttpHeaders.LOCATION.toString());
if (StringUtils.isBlank(location)) {
throw new RuntimeException("重定向响应缺少Location头");
}
currentUrl = resolveRedirectUrl(currentUrl, location);
}
throw new RuntimeException("重定向处理失败");
} }
/** /**
@@ -217,15 +281,7 @@ public class JsHttpClient {
*/ */
public JsHttpResponse getNoRedirect(String url) { public JsHttpResponse getNoRedirect(String url) {
validateUrlSecurity(url); validateUrlSecurity(url);
return executeRequest(() -> { return executeRequest(HttpMethod.GET, url, null, false);
HttpRequest<Buffer> request = client.getAbs(url);
if (!headers.isEmpty()) {
request.putHeaders(headers);
}
// 设置不跟随重定向
request.followRedirects(false);
return request.send();
});
} }
/** /**
@@ -236,26 +292,7 @@ public class JsHttpClient {
*/ */
public JsHttpResponse post(String url, Object data) { public JsHttpResponse post(String url, Object data) {
validateUrlSecurity(url); validateUrlSecurity(url);
return executeRequest(() -> { return executeRequest(HttpMethod.POST, url, bodyFromData(data), false);
HttpRequest<Buffer> request = client.postAbs(url);
if (!headers.isEmpty()) {
request.putHeaders(headers);
}
if (data != null) {
if (data instanceof String) {
return request.sendBuffer(Buffer.buffer((String) data));
} else if (data instanceof Map) {
@SuppressWarnings("unchecked")
Map<String, String> mapData = (Map<String, String>) data;
return request.sendForm(MultiMap.caseInsensitiveMultiMap().addAll(mapData));
} else {
return request.sendJson(data);
}
} else {
return request.send();
}
});
} }
/** /**
@@ -266,26 +303,7 @@ public class JsHttpClient {
*/ */
public JsHttpResponse put(String url, Object data) { public JsHttpResponse put(String url, Object data) {
validateUrlSecurity(url); validateUrlSecurity(url);
return executeRequest(() -> { return executeRequest(HttpMethod.PUT, url, bodyFromData(data), false);
HttpRequest<Buffer> request = client.putAbs(url);
if (!headers.isEmpty()) {
request.putHeaders(headers);
}
if (data != null) {
if (data instanceof String) {
return request.sendBuffer(Buffer.buffer((String) data));
} else if (data instanceof Map) {
@SuppressWarnings("unchecked")
Map<String, String> mapData = (Map<String, String>) data;
return request.sendForm(MultiMap.caseInsensitiveMultiMap().addAll(mapData));
} else {
return request.sendJson(data);
}
} else {
return request.send();
}
});
} }
/** /**
@@ -294,13 +312,8 @@ public class JsHttpClient {
* @return HTTP响应 * @return HTTP响应
*/ */
public JsHttpResponse delete(String url) { public JsHttpResponse delete(String url) {
return executeRequest(() -> { validateUrlSecurity(url);
HttpRequest<Buffer> request = client.deleteAbs(url); return executeRequest(HttpMethod.DELETE, url, null, false);
if (!headers.isEmpty()) {
request.putHeaders(headers);
}
return request.send();
});
} }
/** /**
@@ -310,26 +323,8 @@ public class JsHttpClient {
* @return HTTP响应 * @return HTTP响应
*/ */
public JsHttpResponse patch(String url, Object data) { public JsHttpResponse patch(String url, Object data) {
return executeRequest(() -> { validateUrlSecurity(url);
HttpRequest<Buffer> request = client.patchAbs(url); return executeRequest(HttpMethod.PATCH, url, bodyFromData(data), false);
if (!headers.isEmpty()) {
request.putHeaders(headers);
}
if (data != null) {
if (data instanceof String) {
return request.sendBuffer(Buffer.buffer((String) data));
} else if (data instanceof Map) {
@SuppressWarnings("unchecked")
Map<String, String> mapData = (Map<String, String>) data;
return request.sendForm(MultiMap.caseInsensitiveMultiMap().addAll(mapData));
} else {
return request.sendJson(data);
}
} else {
return request.send();
}
});
} }
/** /**
@@ -340,6 +335,12 @@ public class JsHttpClient {
*/ */
public JsHttpClient putHeader(String name, String value) { public JsHttpClient putHeader(String name, String value) {
if (name != null && value != null) { if (name != null && value != null) {
if (headers.size() >= MAX_HEADER_COUNT && !headers.contains(name)) {
throw new IllegalArgumentException("请求头数量超过限制");
}
if (value.length() > MAX_HEADER_VALUE_LENGTH) {
throw new IllegalArgumentException("请求头过长: " + name);
}
headers.set(name, value); headers.set(name, value);
} }
return this; return this;
@@ -353,9 +354,7 @@ public class JsHttpClient {
public JsHttpClient putHeaders(Map<String, String> headersMap) { public JsHttpClient putHeaders(Map<String, String> headersMap) {
if (headersMap != null) { if (headersMap != null) {
for (Map.Entry<String, String> entry : headersMap.entrySet()) { for (Map.Entry<String, String> entry : headersMap.entrySet()) {
if (entry.getKey() != null && entry.getValue() != null) { putHeader(entry.getKey(), entry.getValue());
headers.set(entry.getKey(), entry.getValue());
}
} }
} }
return this; return this;
@@ -380,7 +379,7 @@ public class JsHttpClient {
public JsHttpClient clearHeaders() { public JsHttpClient clearHeaders() {
headers.clear(); headers.clear();
// 重新设置默认头 // 重新设置默认头
headers.set("Accept-Encoding", "gzip, deflate, br, zstd"); headers.set("Accept-Encoding", DEFAULT_ACCEPT_ENCODING);
headers.set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 Edg/140.0.0.0"); headers.set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 Edg/140.0.0.0");
headers.set("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6"); headers.set("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6");
return this; return this;
@@ -405,7 +404,7 @@ public class JsHttpClient {
*/ */
public JsHttpClient setTimeout(int seconds) { public JsHttpClient setTimeout(int seconds) {
if (seconds > 0) { if (seconds > 0) {
this.timeoutSeconds = seconds; this.timeoutSeconds = Math.min(seconds, MAX_TIMEOUT_SECONDS);
} }
return this; return this;
} }
@@ -450,19 +449,12 @@ public class JsHttpClient {
* @return HTTP响应 * @return HTTP响应
*/ */
public JsHttpResponse sendForm(Map<String, String> data) { public JsHttpResponse sendForm(Map<String, String> data) {
return executeRequest(() -> { throw new IllegalArgumentException("sendForm(data) 缺少请求URL,请使用 post(url, data)");
HttpRequest<Buffer> request = client.postAbs("");
if (!headers.isEmpty()) {
request.putHeaders(headers);
} }
MultiMap formData = MultiMap.caseInsensitiveMultiMap(); public JsHttpResponse sendForm(String url, Map<String, String> data) {
if (data != null) { validateUrlSecurity(url);
formData.addAll(data); return executeRequest(HttpMethod.POST, url, formBody(data), false);
}
return request.sendForm(formData);
});
} }
/** /**
@@ -474,34 +466,8 @@ public class JsHttpClient {
* @return HTTP响应 * @return HTTP响应
*/ */
public JsHttpResponse sendMultipartForm(String url, Map<String, Object> data) { public JsHttpResponse sendMultipartForm(String url, Map<String, Object> data) {
return executeRequest(() -> { validateUrlSecurity(url);
HttpRequest<Buffer> request = client.postAbs(url); return executeRequest(HttpMethod.POST, url, multipartBody(data), false);
if (!headers.isEmpty()) {
request.putHeaders(headers);
}
MultipartForm form = MultipartForm.create();
if (data != null) {
for (Map.Entry<String, Object> entry : data.entrySet()) {
String key = entry.getKey();
Object value = entry.getValue();
if (value instanceof String) {
form.attribute(key, (String) value);
} else if (value instanceof byte[]) {
form.binaryFileUpload(key, key, Buffer.buffer((byte[]) value), "application/octet-stream");
} else if (value instanceof Buffer) {
form.binaryFileUpload(key, key, (Buffer) value, "application/octet-stream");
} else if (value != null) {
// 其他类型转换为字符串
form.attribute(key, value.toString());
}
}
}
return request.sendMultipartForm(form);
});
} }
/** /**
@@ -510,44 +476,103 @@ public class JsHttpClient {
* @return HTTP响应 * @return HTTP响应
*/ */
public JsHttpResponse sendJson(Object data) { public JsHttpResponse sendJson(Object data) {
return executeRequest(() -> { throw new IllegalArgumentException("sendJson(data) 缺少请求URL,请使用 post(url, data)");
HttpRequest<Buffer> request = client.postAbs("");
if (!headers.isEmpty()) {
request.putHeaders(headers);
} }
return request.sendJson(data); public JsHttpResponse sendJson(String url, Object data) {
}); validateUrlSecurity(url);
return executeRequest(HttpMethod.POST, url, jsonBody(data), false);
} }
/** /**
* 执行HTTP请求(同步) * 执行HTTP请求(同步)
*/ */
private JsHttpResponse executeRequest(RequestExecutor executor) { private JsHttpResponse executeRequest(HttpMethod method, String url, RequestBody requestBody, boolean followRedirects) {
try { if (closed.get()) {
Promise<HttpResponse<Buffer>> promise = Promise.promise(); throw new IllegalStateException("HTTP客户端已关闭");
Future<HttpResponse<Buffer>> future = executor.execute();
future.onComplete(result -> {
if (result.succeeded()) {
promise.complete(result.result());
} else {
promise.fail(result.cause());
} }
}).onFailure(Throwable::printStackTrace); AtomicReference<HttpClientRequest> requestRef = new AtomicReference<>();
AtomicBoolean abandoned = new AtomicBoolean(false);
try {
Promise<JsHttpResponse> promise = Promise.promise();
// 等待响应完成(使用配置的超时时间) RequestOptions options = new RequestOptions()
HttpResponse<Buffer> response = promise.future().toCompletionStage() .setMethod(method)
.setAbsoluteURI(url)
.setFollowRedirects(followRedirects)
.setTimeout(TimeUnit.SECONDS.toMillis(timeoutSeconds))
.setHeaders(MultiMap.caseInsensitiveMultiMap().setAll(headers));
client.request(options).onComplete(ar -> {
if (ar.failed()) {
promise.tryFail(ar.cause());
return;
}
HttpClientRequest request = ar.result();
synchronized (requestLock) {
if (closed.get() || abandoned.get()) {
request.reset();
promise.tryFail("HTTP客户端已关闭");
return;
}
activeRequests.add(request);
requestRef.set(request);
request.exceptionHandler(e -> {
finishRequest(request);
promise.tryFail(e);
});
request.response().onComplete(responseAr -> {
if (responseAr.succeeded()) {
collectResponse(request, responseAr.result(), promise);
} else {
finishRequest(request);
promise.tryFail(responseAr.cause());
}
});
if (closed.get() || abandoned.get()) {
request.reset();
finishRequest(request);
promise.tryFail("HTTP客户端已关闭");
return;
}
if (requestBody == null || requestBody.body() == null) {
request.end().onFailure(e -> {
finishRequest(request);
promise.tryFail(e);
});
} else {
request.headers().set(HttpHeaders.CONTENT_LENGTH, String.valueOf(requestBody.body().length()));
if (StringUtils.isNotEmpty(requestBody.contentType())) {
request.headers().set(HttpHeaders.CONTENT_TYPE, requestBody.contentType());
}
request.end(requestBody.body()).onFailure(e -> {
finishRequest(request);
promise.tryFail(e);
});
}
}
});
return promise.future().toCompletionStage()
.toCompletableFuture() .toCompletableFuture()
.get(timeoutSeconds, TimeUnit.SECONDS); .get(timeoutSeconds, TimeUnit.SECONDS);
return new JsHttpResponse(response);
} catch (TimeoutException e) { } catch (TimeoutException e) {
// RequestOptions timeout 通常会先触发;这里再兜底,避免等待线程返回后请求还在后台下载。
String errorMsg = "HTTP请求超时(" + timeoutSeconds + "秒)"; String errorMsg = "HTTP请求超时(" + timeoutSeconds + "秒)";
abandoned.set(true);
synchronized (requestLock) {
abortRequest(requestRef);
}
log.error(errorMsg, e); log.error(errorMsg, e);
throw new RuntimeException(errorMsg, e); throw new RuntimeException(errorMsg, e);
} catch (Exception e) { } catch (Exception e) {
abandoned.set(true);
synchronized (requestLock) {
abortRequest(requestRef);
}
String errorMsg = e.getMessage(); String errorMsg = e.getMessage();
if (errorMsg == null || errorMsg.trim().isEmpty()) { if (errorMsg == null || errorMsg.trim().isEmpty()) {
errorMsg = e.getClass().getSimpleName(); errorMsg = e.getClass().getSimpleName();
@@ -560,12 +585,195 @@ public class JsHttpClient {
} }
} }
/** private static boolean isRedirectStatus(int statusCode) {
* 请求执行器接口 return statusCode == 301 || statusCode == 302 || statusCode == 303
*/ || statusCode == 307 || statusCode == 308;
@FunctionalInterface }
private interface RequestExecutor {
Future<HttpResponse<Buffer>> execute(); private String resolveRedirectUrl(String currentUrl, String location) {
try {
URI redirectUri = new URI(currentUrl).resolve(location.trim());
String scheme = redirectUri.getScheme();
if (!"http".equalsIgnoreCase(scheme) && !"https".equalsIgnoreCase(scheme)) {
throw new SecurityException("🔒 安全拦截: 重定向协议不被允许");
}
String redirectUrl = redirectUri.toString();
validateUrlSecurity(redirectUrl);
return redirectUrl;
} catch (SecurityException e) {
throw e;
} catch (Exception e) {
throw new RuntimeException("解析重定向地址失败: " + e.getMessage(), e);
}
}
private void collectResponse(HttpClientRequest request, HttpClientResponse response, Promise<JsHttpResponse> promise) {
Buffer body = Buffer.buffer();
AtomicBoolean done = new AtomicBoolean(false);
String contentLengthHeader = response.getHeader(HttpHeaders.CONTENT_LENGTH.toString());
if (StringUtils.isNumeric(contentLengthHeader)) {
long contentLength = Long.parseLong(contentLengthHeader);
if (contentLength > MAX_RESPONSE_BODY_BYTES) {
done.set(true);
request.reset();
finishRequest(request);
promise.tryFail("响应体过大: " + contentLength + " bytes");
return;
}
}
response.exceptionHandler(e -> {
if (done.compareAndSet(false, true)) {
finishRequest(request);
promise.tryFail(e);
}
});
response.handler(chunk -> {
if (done.get()) {
return;
}
if (body.length() + chunk.length() > MAX_RESPONSE_BODY_BYTES) {
if (done.compareAndSet(false, true)) {
request.reset();
finishRequest(request);
promise.tryFail("响应体过大: " + (body.length() + chunk.length()) + " bytes");
}
return;
}
body.appendBuffer(chunk);
});
response.endHandler(v -> {
if (done.compareAndSet(false, true)) {
finishRequest(request);
promise.tryComplete(new JsHttpResponse(
response.statusCode(),
MultiMap.caseInsensitiveMultiMap().setAll(response.headers()),
body,
response.statusMessage(),
null
));
}
});
response.resume();
}
private void finishRequest(HttpClientRequest request) {
if (request != null) {
activeRequests.remove(request);
}
}
private void abortRequest(AtomicReference<HttpClientRequest> requestRef) {
HttpClientRequest request = requestRef.get();
if (request != null) {
try {
request.reset();
} finally {
finishRequest(request);
}
}
}
private RequestBody bodyFromData(Object data) {
if (data == null) {
return null;
}
if (data instanceof String str) {
return plainTextBody(str);
}
if (data instanceof Buffer buffer) {
return limitedBody(buffer, null);
}
if (data instanceof byte[] bytes) {
return limitedBody(Buffer.buffer(bytes), null);
}
if (data instanceof Map<?, ?> map) {
Map<String, String> formMap = new HashMap<>();
map.forEach((key, value) -> {
if (key != null && value != null) {
formMap.put(String.valueOf(key), String.valueOf(value));
}
});
return formBody(formMap);
}
return jsonBody(data);
}
private RequestBody plainTextBody(String data) {
return limitedBody(Buffer.buffer(data, StandardCharsets.UTF_8.name()), null);
}
private RequestBody jsonBody(Object data) {
Buffer body = data == null ? Buffer.buffer() : Buffer.buffer(Json.encode(data), StandardCharsets.UTF_8.name());
return limitedBody(body, "application/json; charset=utf-8");
}
private RequestBody formBody(Map<String, String> data) {
StringBuilder encoded = new StringBuilder();
if (data != null) {
for (Map.Entry<String, String> entry : data.entrySet()) {
if (encoded.length() > 0) {
encoded.append('&');
}
encoded.append(urlEncode(entry.getKey()));
encoded.append('=');
encoded.append(urlEncode(entry.getValue()));
}
}
return limitedBody(Buffer.buffer(encoded.toString(), StandardCharsets.UTF_8.name()),
"application/x-www-form-urlencoded; charset=utf-8");
}
private RequestBody multipartBody(Map<String, Object> data) {
String boundary = "----NetdiskJsHttpClientBoundary" + UUID.randomUUID().toString().replace("-", "");
Buffer body = Buffer.buffer();
if (data != null) {
for (Map.Entry<String, Object> entry : data.entrySet()) {
String key = entry.getKey();
Object value = entry.getValue();
if (key == null || value == null) {
continue;
}
appendAscii(body, "--" + boundary + "\r\n");
if (value instanceof byte[] bytes) {
appendAscii(body, "Content-Disposition: form-data; name=\"" + escapeMultipart(key)
+ "\"; filename=\"" + escapeMultipart(key) + "\"\r\n");
appendAscii(body, "Content-Type: application/octet-stream\r\n\r\n");
body.appendBytes(bytes);
appendAscii(body, "\r\n");
} else if (value instanceof Buffer buffer) {
appendAscii(body, "Content-Disposition: form-data; name=\"" + escapeMultipart(key)
+ "\"; filename=\"" + escapeMultipart(key) + "\"\r\n");
appendAscii(body, "Content-Type: application/octet-stream\r\n\r\n");
body.appendBuffer(buffer);
appendAscii(body, "\r\n");
} else {
appendAscii(body, "Content-Disposition: form-data; name=\"" + escapeMultipart(key) + "\"\r\n\r\n");
body.appendString(String.valueOf(value), StandardCharsets.UTF_8.name());
appendAscii(body, "\r\n");
}
ensureRequestBodyLimit(body);
}
}
appendAscii(body, "--" + boundary + "--\r\n");
return limitedBody(body, "multipart/form-data; boundary=" + boundary);
}
private static void appendAscii(Buffer body, String value) {
body.appendString(value, StandardCharsets.US_ASCII.name());
}
private static String escapeMultipart(String value) {
return value.replace("\\", "\\\\").replace("\"", "\\\"");
}
private static RequestBody limitedBody(Buffer body, String contentType) {
ensureRequestBodyLimit(body);
return new RequestBody(body, contentType);
}
private record RequestBody(Buffer body, String contentType) {
} }
/** /**
@@ -573,10 +781,29 @@ public class JsHttpClient {
*/ */
public static class JsHttpResponse { public static class JsHttpResponse {
private final HttpResponse<Buffer> response; private final int statusCode;
private final MultiMap headers;
private final Buffer body;
private final String statusMessage;
private final HttpResponse<Buffer> originalResponse;
public JsHttpResponse(HttpResponse<Buffer> response) { public JsHttpResponse(HttpResponse<Buffer> response) {
this.response = response; this(
response.statusCode(),
MultiMap.caseInsensitiveMultiMap().setAll(response.headers()),
response.body(),
response.statusMessage(),
response
);
}
public JsHttpResponse(int statusCode, MultiMap headers, Buffer body, String statusMessage,
HttpResponse<Buffer> originalResponse) {
this.statusCode = statusCode;
this.headers = headers == null ? MultiMap.caseInsensitiveMultiMap() : headers;
this.body = body == null ? Buffer.buffer() : body;
this.statusMessage = statusMessage;
this.originalResponse = originalResponse;
} }
/** /**
@@ -584,7 +811,7 @@ public class JsHttpClient {
* @return 响应体字符串 * @return 响应体字符串
*/ */
public String body() { public String body() {
return HttpResponseHelper.asText(response); return HttpResponseHelper.asText(body, header(HttpHeaders.CONTENT_ENCODING.toString()));
} }
/** /**
@@ -593,7 +820,7 @@ public class JsHttpClient {
*/ */
public Object json() { public Object json() {
try { try {
JsonObject jsonObject = HttpResponseHelper.asJson(response); JsonObject jsonObject = HttpResponseHelper.asJson(body, header(HttpHeaders.CONTENT_ENCODING.toString()));
if (jsonObject == null || jsonObject.isEmpty()) { if (jsonObject == null || jsonObject.isEmpty()) {
return null; return null;
} }
@@ -611,7 +838,7 @@ public class JsHttpClient {
* @return 状态码 * @return 状态码
*/ */
public int statusCode() { public int statusCode() {
return response.statusCode(); return statusCode;
} }
/** /**
@@ -620,7 +847,7 @@ public class JsHttpClient {
* @return 头值 * @return 头值
*/ */
public String header(String name) { public String header(String name) {
return response.getHeader(name); return headers.get(name);
} }
/** /**
@@ -628,10 +855,9 @@ public class JsHttpClient {
* @return 响应头Map * @return 响应头Map
*/ */
public Map<String, String> headers() { public Map<String, String> headers() {
MultiMap responseHeaders = response.headers();
Map<String, String> result = new HashMap<>(); Map<String, String> result = new HashMap<>();
for (String name : responseHeaders.names()) { for (String name : headers.names()) {
result.put(name, responseHeaders.get(name)); result.put(name, headers.get(name));
} }
return result; return result;
} }
@@ -649,8 +875,14 @@ public class JsHttpClient {
* 获取原始响应对象 * 获取原始响应对象
* @return HttpResponse对象 * @return HttpResponse对象
*/ */
@Deprecated
public HttpResponse<Buffer> getOriginalResponse() { public HttpResponse<Buffer> getOriginalResponse() {
return response; if (originalResponse == null) {
throw new UnsupportedOperationException(
"流式HTTP客户端不再保留原始Vert.x HttpResponse,请使用statusCode/header/headers/body/bodyBytes方法"
);
}
return originalResponse;
} }
/** /**
@@ -658,11 +890,8 @@ public class JsHttpClient {
* @return 响应体字节数组 * @return 响应体字节数组
*/ */
public byte[] bodyBytes() { public byte[] bodyBytes() {
Buffer buffer = response.body(); ensureResponseBodyLimit(body);
if (buffer == null) { return body.getBytes();
return new byte[0];
}
return buffer.getBytes();
} }
/** /**
@@ -670,11 +899,46 @@ public class JsHttpClient {
* @return 响应体大小(字节) * @return 响应体大小(字节)
*/ */
public long bodySize() { public long bodySize() {
Buffer buffer = response.body(); return body.length();
if (buffer == null) {
return 0;
} }
return buffer.length();
public String statusMessage() {
return statusMessage;
}
}
/**
* 关闭 HttpClient 释放连接池资源
* 仅关闭自建的 client(代理模式),共享实例不关闭
*/
public void close() {
if (!closed.compareAndSet(false, true)) {
return;
}
synchronized (requestLock) {
for (HttpClientRequest request : activeRequests) {
try {
request.reset();
} catch (Exception e) {
log.debug("重置 JavaScript HTTP 请求失败: {}", e.getMessage());
}
}
activeRequests.clear();
}
if (ownClient && client != null) {
client.close();
}
}
private static void ensureResponseBodyLimit(Buffer buffer) {
if (buffer != null && buffer.length() > MAX_RESPONSE_BODY_BYTES) {
throw new IllegalArgumentException("响应体过大: " + buffer.length() + " bytes");
}
}
private static void ensureRequestBodyLimit(Buffer buffer) {
if (buffer != null && buffer.length() > MAX_REQUEST_BODY_BYTES) {
throw new IllegalArgumentException("请求体过大: " + buffer.length() + " bytes");
} }
} }
} }
@@ -6,6 +6,7 @@ import cn.qaiu.entity.ShareLinkInfo;
import cn.qaiu.parser.IPanTool; import cn.qaiu.parser.IPanTool;
import cn.qaiu.parser.custom.CustomParserConfig; import cn.qaiu.parser.custom.CustomParserConfig;
import io.vertx.core.Future; import io.vertx.core.Future;
import io.vertx.core.Promise;
import io.vertx.core.WorkerExecutor; import io.vertx.core.WorkerExecutor;
import io.vertx.core.json.JsonObject; import io.vertx.core.json.JsonObject;
import org.openjdk.nashorn.api.scripting.NashornScriptEngineFactory; import org.openjdk.nashorn.api.scripting.NashornScriptEngineFactory;
@@ -20,6 +21,13 @@ import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.List; import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.stream.Collectors; import java.util.stream.Collectors;
/** /**
@@ -29,21 +37,46 @@ import java.util.stream.Collectors;
* @author <a href="https://qaiu.top">QAIU</a> * @author <a href="https://qaiu.top">QAIU</a>
* Create at 2025/10/17 * Create at 2025/10/17
*/ */
public class JsParserExecutor implements IPanTool { public class JsParserExecutor implements IPanTool, AutoCloseable {
private static final Logger log = LoggerFactory.getLogger(JsParserExecutor.class); private static final Logger log = LoggerFactory.getLogger(JsParserExecutor.class);
private static final WorkerExecutor EXECUTOR = WebClientVertxInit.get().createSharedWorkerExecutor("parser-executor", 32); private static volatile WorkerExecutor EXECUTOR;
private static final Object EXECUTOR_LOCK = new Object();
private static volatile boolean executorShutdown = false;
private static String FETCH_RUNTIME_JS = null; /** 安全网调度器:当 onComplete 未触发时,延迟强制释放资源 */
private static final ScheduledExecutorService CLEANUP_SCHEDULER =
Executors.newSingleThreadScheduledExecutor(r -> {
Thread t = new Thread(r, "js-parser-cleanup-safety");
t.setDaemon(true);
return t;
});
private static final long EXECUTION_TIMEOUT_SECONDS = 30;
private static final int MAX_RESULT_STRING_LENGTH = 1024 * 1024;
private static final int MAX_FILE_LIST_SIZE = 1000;
private static final int MAX_FILE_FIELD_LENGTH = 4096;
private static final int MAX_CONCURRENT_EXECUTIONS =
Math.max(1, Integer.getInteger("parser.custom.js.maxConcurrentExecutions", 32));
private static final Semaphore EXECUTION_PERMITS = new Semaphore(MAX_CONCURRENT_EXECUTIONS);
private static volatile String FETCH_RUNTIME_JS = null;
private final CustomParserConfig config; private final CustomParserConfig config;
private final ShareLinkInfo shareLinkInfo; private final ShareLinkInfo shareLinkInfo;
private final ScriptEngine engine; private volatile ScriptEngine engine;
private final Object engineLock = new Object();
private final JsHttpClient httpClient; private final JsHttpClient httpClient;
private final JsLogger jsLogger; private final JsLogger jsLogger;
private final JsShareLinkInfoWrapper shareLinkInfoWrapper; private final JsShareLinkInfoWrapper shareLinkInfoWrapper;
private final JsFetchBridge fetchBridge; private final JsFetchBridge fetchBridge;
/** 标记是否已释放,防止重复关闭 */
private final AtomicBoolean closed = new AtomicBoolean(false);
private final Object lifecycleLock = new Object();
private volatile boolean running = false;
private volatile boolean closeRequested = false;
/** 安全网定时任务句柄,正常完成时取消 */
private volatile ScheduledFuture<?> safetyCleanupFuture = null;
public JsParserExecutor(ShareLinkInfo shareLinkInfo, CustomParserConfig config) { public JsParserExecutor(ShareLinkInfo shareLinkInfo, CustomParserConfig config) {
this.config = config; this.config = config;
@@ -59,7 +92,6 @@ public class JsParserExecutor implements IPanTool {
this.jsLogger = new JsLogger("JsParser-" + config.getType()); this.jsLogger = new JsLogger("JsParser-" + config.getType());
this.shareLinkInfoWrapper = new JsShareLinkInfoWrapper(shareLinkInfo); this.shareLinkInfoWrapper = new JsShareLinkInfoWrapper(shareLinkInfo);
this.fetchBridge = new JsFetchBridge(httpClient); this.fetchBridge = new JsFetchBridge(httpClient);
this.engine = initEngine();
} }
/** /**
@@ -111,6 +143,7 @@ public class JsParserExecutor implements IPanTool {
if (engine == null) { if (engine == null) {
throw new RuntimeException("无法创建JavaScript引擎,请确保Nashorn可用"); throw new RuntimeException("无法创建JavaScript引擎,请确保Nashorn可用");
} }
this.engine = engine;
// 注入Java对象到JavaScript环境 // 注入Java对象到JavaScript环境
engine.put("http", httpClient); engine.put("http", httpClient);
@@ -146,12 +179,190 @@ public class JsParserExecutor implements IPanTool {
} }
} }
private ScriptEngine engine() {
ScriptEngine current = engine;
if (current != null) {
return current;
}
synchronized (engineLock) {
if (closed.get()) {
throw new IllegalStateException("JavaScript解析器已关闭");
}
if (engine == null) {
engine = initEngine();
}
return engine;
}
}
private void beginExecution() {
synchronized (lifecycleLock) {
if (closed.get() || closeRequested) {
throw new IllegalStateException("JavaScript解析器已关闭");
}
if (running) {
throw new IllegalStateException("JavaScript解析器已在运行");
}
running = true;
}
}
private void finishExecution() {
synchronized (lifecycleLock) {
running = false;
if (closeRequested) {
doClose();
}
}
}
/**
* 释放资源(ScriptEngine 和 HttpClient),避免内存泄漏
* 幂等:可安全多次调用
*/
@Override
public void close() {
synchronized (lifecycleLock) {
closeRequested = true;
cancelSafetyCleanup();
if (running || closed.get()) {
closeExternalResources();
return;
}
doClose();
}
}
private void doClose() {
if (!closed.compareAndSet(false, true)) return;
closeRequested = false;
closeExternalResources();
cleanupEngine();
}
private void closeExternalResources() {
if (httpClient != null) {
httpClient.close();
}
}
private void cleanupEngine() {
// 清除 ScriptEngine 持有的所有引用和内部状态,帮助 GC 回收
if (engine != null) {
try {
engine.put("http", null);
engine.put("logger", null);
engine.put("shareLinkInfo", null);
engine.put("JavaFetch", null);
// 彻底清除 ENGINE_SCOPE bindings,释放 JS AST、编译函数、闭包等运行时状态
var bindings = engine.getBindings(javax.script.ScriptContext.ENGINE_SCOPE);
if (bindings != null) {
bindings.clear();
}
} catch (Exception e) {
log.warn("清理 ScriptEngine bindings 失败: {}", e.getMessage());
}
}
}
private void cancelSafetyCleanup() {
// 取消安全网定时任务(如果正常完成则无需再触发)
if (safetyCleanupFuture != null) {
safetyCleanupFuture.cancel(false);
safetyCleanupFuture = null;
}
}
/**
* 关闭全局 WorkerExecutor 和清理调度器(应在应用关闭时调用)
*/
public static void shutdownExecutor() {
synchronized (EXECUTOR_LOCK) {
executorShutdown = true;
if (EXECUTOR != null) {
EXECUTOR.close();
EXECUTOR = null;
log.info("JsParserExecutor WorkerExecutor 已关闭");
}
}
CLEANUP_SCHEDULER.shutdown();
}
/**
* 获取或创建 WorkerExecutor(懒加载)
*/
private static WorkerExecutor getExecutor() {
synchronized (EXECUTOR_LOCK) {
if (executorShutdown) {
throw new IllegalStateException("JavaScript解析器 WorkerExecutor 已关闭");
}
if (EXECUTOR == null) {
EXECUTOR = WebClientVertxInit.get().createSharedWorkerExecutor("parser-executor", 32);
}
return EXECUTOR;
}
}
private <T> Future<T> executeBlockingWithPermit(String operation, Callable<T> blockingCode) {
if (!EXECUTION_PERMITS.tryAcquire()) {
String message = "JavaScript " + operation + " 执行并发已满,请稍后重试";
jsLogger.error(message);
close();
return Future.failedFuture(message);
}
try {
return getExecutor().executeBlocking(() -> {
boolean executionStarted = false;
try {
beginExecution();
executionStarted = true;
return blockingCode.call();
} finally {
if (executionStarted) {
finishExecution();
}
EXECUTION_PERMITS.release();
}
});
} catch (Throwable e) {
EXECUTION_PERMITS.release();
close();
return Future.failedFuture(e);
}
}
private <T> Future<T> withTimeout(Future<T> executionFuture, String operation) {
Promise<T> promise = Promise.promise();
try {
safetyCleanupFuture = CLEANUP_SCHEDULER.schedule(() -> {
if (promise.tryFail("JavaScript " + operation + " 执行超时(" + EXECUTION_TIMEOUT_SECONDS + "秒)")) {
jsLogger.error("{} 执行超时,已停止外部HTTP资源;ScriptEngine将在执行线程退出后清理", operation);
close();
}
}, EXECUTION_TIMEOUT_SECONDS, TimeUnit.SECONDS);
} catch (Exception e) {
log.warn("安全网调度失败: {}", e.getMessage());
}
executionFuture.onComplete(ar -> {
cancelSafetyCleanup();
if (ar.succeeded()) {
promise.tryComplete(ar.result());
} else {
promise.tryFail(ar.cause());
}
close();
});
return promise.future();
}
@Override @Override
public Future<String> parse() { public Future<String> parse() {
jsLogger.info("开始执行JavaScript解析器: {}", config.getType()); jsLogger.info("开始执行JavaScript解析器: {}", config.getType());
// 使用executeBlocking在工作线程上执行,避免阻塞EventLoop线程 // 使用executeBlocking在工作线程上执行,避免阻塞EventLoop线程
return EXECUTOR.executeBlocking(() -> { Future<String> executionFuture = executeBlockingWithPermit("parse", () -> {
ScriptEngine engine = engine();
// 直接调用全局parse函数 // 直接调用全局parse函数
Object parseFunction = engine.get("parse"); Object parseFunction = engine.get("parse");
if (parseFunction == null) { if (parseFunction == null) {
@@ -163,8 +374,9 @@ public class JsParserExecutor implements IPanTool {
Object result = parseMirror.call(null, shareLinkInfoWrapper, httpClient, jsLogger); Object result = parseMirror.call(null, shareLinkInfoWrapper, httpClient, jsLogger);
if (result instanceof String) { if (result instanceof String) {
jsLogger.info("解析成功: {}", result); String resultText = limitResultString((String) result, "parse");
return (String) result; jsLogger.info("解析成功,结果长度: {}", resultText.length());
return resultText;
} else { } else {
jsLogger.error("parse方法返回值类型错误,期望String,实际: {}", jsLogger.error("parse方法返回值类型错误,期望String,实际: {}",
result != null ? result.getClass().getSimpleName() : "null"); result != null ? result.getClass().getSimpleName() : "null");
@@ -174,6 +386,7 @@ public class JsParserExecutor implements IPanTool {
throw new RuntimeException("parse函数类型错误"); throw new RuntimeException("parse函数类型错误");
} }
}); });
return withTimeout(executionFuture, "parse");
} }
@Override @Override
@@ -181,7 +394,8 @@ public class JsParserExecutor implements IPanTool {
jsLogger.info("开始执行JavaScript文件列表解析: {}", config.getType()); jsLogger.info("开始执行JavaScript文件列表解析: {}", config.getType());
// 使用executeBlocking在工作线程上执行,避免阻塞EventLoop线程 // 使用executeBlocking在工作线程上执行,避免阻塞EventLoop线程
return EXECUTOR.executeBlocking(() -> { Future<List<FileInfo>> executionFuture = executeBlockingWithPermit("parseFileList", () -> {
ScriptEngine engine = engine();
// 直接调用全局parseFileList函数 // 直接调用全局parseFileList函数
Object parseFileListFunction = engine.get("parseFileList"); Object parseFileListFunction = engine.get("parseFileList");
if (parseFileListFunction == null) { if (parseFileListFunction == null) {
@@ -207,6 +421,7 @@ public class JsParserExecutor implements IPanTool {
throw new RuntimeException("parseFileList函数类型错误"); throw new RuntimeException("parseFileList函数类型错误");
} }
}); });
return withTimeout(executionFuture, "parseFileList");
} }
@Override @Override
@@ -214,7 +429,8 @@ public class JsParserExecutor implements IPanTool {
jsLogger.info("开始执行JavaScript按ID解析: {}", config.getType()); jsLogger.info("开始执行JavaScript按ID解析: {}", config.getType());
// 使用executeBlocking在工作线程上执行,避免阻塞EventLoop线程 // 使用executeBlocking在工作线程上执行,避免阻塞EventLoop线程
return EXECUTOR.executeBlocking(() -> { Future<String> executionFuture = executeBlockingWithPermit("parseById", () -> {
ScriptEngine engine = engine();
// 直接调用全局parseById函数 // 直接调用全局parseById函数
Object parseByIdFunction = engine.get("parseById"); Object parseByIdFunction = engine.get("parseById");
if (parseByIdFunction == null) { if (parseByIdFunction == null) {
@@ -227,8 +443,9 @@ public class JsParserExecutor implements IPanTool {
Object result = parseByIdMirror.call(null, shareLinkInfoWrapper, httpClient, jsLogger); Object result = parseByIdMirror.call(null, shareLinkInfoWrapper, httpClient, jsLogger);
if (result instanceof String) { if (result instanceof String) {
jsLogger.info("按ID解析成功: {}", result); String resultText = limitResultString((String) result, "parseById");
return (String) result; jsLogger.info("按ID解析成功,结果长度: {}", resultText.length());
return resultText;
} else { } else {
jsLogger.error("parseById方法返回值类型错误,期望String,实际: {}", jsLogger.error("parseById方法返回值类型错误,期望String,实际: {}",
result != null ? result.getClass().getSimpleName() : "null"); result != null ? result.getClass().getSimpleName() : "null");
@@ -238,6 +455,7 @@ public class JsParserExecutor implements IPanTool {
throw new RuntimeException("parseById函数类型错误"); throw new RuntimeException("parseById函数类型错误");
} }
}); });
return withTimeout(executionFuture, "parseById");
} }
/** /**
@@ -247,6 +465,9 @@ public class JsParserExecutor implements IPanTool {
List<FileInfo> fileList = new ArrayList<>(); List<FileInfo> fileList = new ArrayList<>();
if (resultMirror.isArray()) { if (resultMirror.isArray()) {
if (resultMirror.size() > MAX_FILE_LIST_SIZE) {
throw new RuntimeException("文件列表数量超过限制: " + resultMirror.size());
}
for (int i = 0; i < resultMirror.size(); i++) { for (int i = 0; i < resultMirror.size(); i++) {
Object item = resultMirror.get(String.valueOf(i)); Object item = resultMirror.get(String.valueOf(i));
if (item instanceof ScriptObjectMirror) { if (item instanceof ScriptObjectMirror) {
@@ -270,13 +491,13 @@ public class JsParserExecutor implements IPanTool {
// 设置基本字段 // 设置基本字段
if (itemMirror.hasMember("fileName")) { if (itemMirror.hasMember("fileName")) {
fileInfo.setFileName(itemMirror.getMember("fileName").toString()); fileInfo.setFileName(limitField(itemMirror.getMember("fileName")));
} }
if (itemMirror.hasMember("fileId")) { if (itemMirror.hasMember("fileId")) {
fileInfo.setFileId(itemMirror.getMember("fileId").toString()); fileInfo.setFileId(limitField(itemMirror.getMember("fileId")));
} }
if (itemMirror.hasMember("fileType")) { if (itemMirror.hasMember("fileType")) {
fileInfo.setFileType(itemMirror.getMember("fileType").toString()); fileInfo.setFileType(limitField(itemMirror.getMember("fileType")));
} }
if (itemMirror.hasMember("size")) { if (itemMirror.hasMember("size")) {
Object size = itemMirror.getMember("size"); Object size = itemMirror.getMember("size");
@@ -285,16 +506,16 @@ public class JsParserExecutor implements IPanTool {
} }
} }
if (itemMirror.hasMember("sizeStr")) { if (itemMirror.hasMember("sizeStr")) {
fileInfo.setSizeStr(itemMirror.getMember("sizeStr").toString()); fileInfo.setSizeStr(limitField(itemMirror.getMember("sizeStr")));
} }
if (itemMirror.hasMember("createTime")) { if (itemMirror.hasMember("createTime")) {
fileInfo.setCreateTime(itemMirror.getMember("createTime").toString()); fileInfo.setCreateTime(limitField(itemMirror.getMember("createTime")));
} }
if (itemMirror.hasMember("updateTime")) { if (itemMirror.hasMember("updateTime")) {
fileInfo.setUpdateTime(itemMirror.getMember("updateTime").toString()); fileInfo.setUpdateTime(limitField(itemMirror.getMember("updateTime")));
} }
if (itemMirror.hasMember("createBy")) { if (itemMirror.hasMember("createBy")) {
fileInfo.setCreateBy(itemMirror.getMember("createBy").toString()); fileInfo.setCreateBy(limitField(itemMirror.getMember("createBy")));
} }
if (itemMirror.hasMember("downloadCount")) { if (itemMirror.hasMember("downloadCount")) {
Object downloadCount = itemMirror.getMember("downloadCount"); Object downloadCount = itemMirror.getMember("downloadCount");
@@ -303,16 +524,16 @@ public class JsParserExecutor implements IPanTool {
} }
} }
if (itemMirror.hasMember("fileIcon")) { if (itemMirror.hasMember("fileIcon")) {
fileInfo.setFileIcon(itemMirror.getMember("fileIcon").toString()); fileInfo.setFileIcon(limitField(itemMirror.getMember("fileIcon")));
} }
if (itemMirror.hasMember("panType")) { if (itemMirror.hasMember("panType")) {
fileInfo.setPanType(itemMirror.getMember("panType").toString()); fileInfo.setPanType(limitField(itemMirror.getMember("panType")));
} }
if (itemMirror.hasMember("parserUrl")) { if (itemMirror.hasMember("parserUrl")) {
fileInfo.setParserUrl(itemMirror.getMember("parserUrl").toString()); fileInfo.setParserUrl(limitField(itemMirror.getMember("parserUrl")));
} }
if (itemMirror.hasMember("previewUrl")) { if (itemMirror.hasMember("previewUrl")) {
fileInfo.setPreviewUrl(itemMirror.getMember("previewUrl").toString()); fileInfo.setPreviewUrl(limitField(itemMirror.getMember("previewUrl")));
} }
return fileInfo; return fileInfo;
@@ -322,4 +543,22 @@ public class JsParserExecutor implements IPanTool {
return null; return null;
} }
} }
private static String limitResultString(String value, String operation) {
if (value.length() > MAX_RESULT_STRING_LENGTH) {
throw new RuntimeException(operation + " 返回结果过大: " + value.length() + " 字符");
}
return value;
}
private static String limitField(Object value) {
if (value == null) {
return null;
}
String text = value.toString();
if (text.length() > MAX_FILE_FIELD_LENGTH) {
throw new RuntimeException("文件字段过长: " + text.length() + " 字符");
}
return text;
}
} }
@@ -21,20 +21,37 @@ import java.util.concurrent.*;
* *
* @author <a href="https://qaiu.top">QAIU</a> * @author <a href="https://qaiu.top">QAIU</a>
*/ */
public class JsPlaygroundExecutor { public class JsPlaygroundExecutor implements AutoCloseable {
private static final Logger log = LoggerFactory.getLogger(JsPlaygroundExecutor.class); private static final Logger log = LoggerFactory.getLogger(JsPlaygroundExecutor.class);
// JavaScript执行超时时间(秒) // JavaScript执行超时时间(秒)
private static final long EXECUTION_TIMEOUT_SECONDS = 30; private static final long EXECUTION_TIMEOUT_SECONDS = 30;
private static final int MAX_RESULT_STRING_LENGTH = 1024 * 1024;
private static final int MAX_FILE_LIST_SIZE = 1000;
private static final int MAX_FILE_FIELD_LENGTH = 4096;
private static final int TIMEOUT_LOG_RETAIN = 50;
// 使用独立的线程池,不受Vert.x的BlockedThreadChecker监控 // 使用有界线程池,防止线程无限增长导致内存溢出
private static final ExecutorService INDEPENDENT_EXECUTOR = Executors.newCachedThreadPool(r -> { private static final int POOL_MAX_THREADS = 16;
private static final int POOL_QUEUE_CAPACITY = 256;
private static final ExecutorService INDEPENDENT_EXECUTOR = new ThreadPoolExecutor(
4, POOL_MAX_THREADS, 60L, TimeUnit.SECONDS,
new ArrayBlockingQueue<>(POOL_QUEUE_CAPACITY),
r -> {
Thread thread = new Thread(r); Thread thread = new Thread(r);
thread.setName("playground-independent-" + System.currentTimeMillis()); thread.setName("playground-independent-" + thread.getId());
thread.setDaemon(true); // 设置为守护线程,服务关闭时自动清理 thread.setDaemon(true);
return thread; return thread;
}); },
(r, executor) -> {
// 拒绝策略:记录日志并抛出异常,避免阻塞 Vert.x EventLoop
log.warn("演练场线程池已满,拒绝任务。活跃线程: {}, 队列大小: {}",
((ThreadPoolExecutor) executor).getActiveCount(),
((ThreadPoolExecutor) executor).getQueue().size());
throw new java.util.concurrent.RejectedExecutionException("演练场线程池已满,请稍后重试");
}
);
// 超时调度线程池,用于处理超时中断 // 超时调度线程池,用于处理超时中断
private static final ScheduledExecutorService TIMEOUT_SCHEDULER = Executors.newScheduledThreadPool(2, r -> { private static final ScheduledExecutorService TIMEOUT_SCHEDULER = Executors.newScheduledThreadPool(2, r -> {
@@ -44,13 +61,28 @@ public class JsPlaygroundExecutor {
return thread; return thread;
}); });
/**
* 关闭静态线程池(应在应用关闭时调用)
*/
public static void shutdownPools() {
INDEPENDENT_EXECUTOR.shutdown();
TIMEOUT_SCHEDULER.shutdown();
log.info("JsPlaygroundExecutor 线程池已关闭");
}
private final ShareLinkInfo shareLinkInfo; private final ShareLinkInfo shareLinkInfo;
private final String jsCode; private final String jsCode;
private final ScriptEngine engine; private volatile ScriptEngine engine;
private final Object engineLock = new Object();
private final JsHttpClient httpClient; private final JsHttpClient httpClient;
private final JsPlaygroundLogger playgroundLogger; private final JsPlaygroundLogger playgroundLogger;
private final JsShareLinkInfoWrapper shareLinkInfoWrapper; private final JsShareLinkInfoWrapper shareLinkInfoWrapper;
private final JsFetchBridge fetchBridge; private final JsFetchBridge fetchBridge;
/** 标记是否已释放,防止重复关闭 */
private volatile boolean closed = false;
private final Object lifecycleLock = new Object();
private volatile boolean running = false;
private volatile boolean closeRequested = false;
/** /**
* 创建演练场执行器 * 创建演练场执行器
@@ -72,7 +104,6 @@ public class JsPlaygroundExecutor {
this.playgroundLogger = new JsPlaygroundLogger(); this.playgroundLogger = new JsPlaygroundLogger();
this.shareLinkInfoWrapper = new JsShareLinkInfoWrapper(shareLinkInfo); this.shareLinkInfoWrapper = new JsShareLinkInfoWrapper(shareLinkInfo);
this.fetchBridge = new JsFetchBridge(httpClient); this.fetchBridge = new JsFetchBridge(httpClient);
this.engine = initEngine();
} }
/** /**
@@ -89,6 +120,7 @@ public class JsPlaygroundExecutor {
if (engine == null) { if (engine == null) {
throw new RuntimeException("无法创建JavaScript引擎,请确保Nashorn可用"); throw new RuntimeException("无法创建JavaScript引擎,请确保Nashorn可用");
} }
this.engine = engine;
// 注入Java对象到JavaScript环境 // 注入Java对象到JavaScript环境
engine.put("http", httpClient); engine.put("http", httpClient);
@@ -134,8 +166,13 @@ public class JsPlaygroundExecutor {
public Future<String> executeParseAsync() { public Future<String> executeParseAsync() {
Promise<String> promise = Promise.promise(); Promise<String> promise = Promise.promise();
final CompletableFuture<String> executionFuture;
try {
// 使用独立的ExecutorService执行,避免Vert.x的BlockedThreadChecker输出警告 // 使用独立的ExecutorService执行,避免Vert.x的BlockedThreadChecker输出警告
CompletableFuture<String> executionFuture = CompletableFuture.supplyAsync(() -> { executionFuture = CompletableFuture.supplyAsync(() -> {
beginExecution();
try {
ScriptEngine engine = engine();
playgroundLogger.infoJava("开始执行parse方法"); playgroundLogger.infoJava("开始执行parse方法");
try { try {
Object parseFunction = engine.get("parse"); Object parseFunction = engine.get("parse");
@@ -151,8 +188,9 @@ public class JsPlaygroundExecutor {
log.debug("[JsPlaygroundExecutor] parse函数执行完成,当前日志数量: {}", playgroundLogger.size()); log.debug("[JsPlaygroundExecutor] parse函数执行完成,当前日志数量: {}", playgroundLogger.size());
if (result instanceof String) { if (result instanceof String) {
playgroundLogger.infoJava("解析成功,返回结果: " + result); String resultText = limitResultString((String) result, "parse");
return (String) result; playgroundLogger.infoJava("解析成功,返回结果长度: " + resultText.length());
return resultText;
} else { } else {
String errorMsg = "parse方法返回值类型错误,期望String,实际: " + String errorMsg = "parse方法返回值类型错误,期望String,实际: " +
(result != null ? result.getClass().getSimpleName() : "null"); (result != null ? result.getClass().getSimpleName() : "null");
@@ -167,16 +205,18 @@ public class JsPlaygroundExecutor {
playgroundLogger.errorJava("执行parse方法失败: " + e.getMessage(), e); playgroundLogger.errorJava("执行parse方法失败: " + e.getMessage(), e);
throw new RuntimeException(e); throw new RuntimeException(e);
} }
}, INDEPENDENT_EXECUTOR); } finally {
finishExecution();
// 创建超时任务,强制取消执行
ScheduledFuture<?> timeoutTask = TIMEOUT_SCHEDULER.schedule(() -> {
if (!executionFuture.isDone()) {
executionFuture.cancel(true); // 强制中断执行线程
playgroundLogger.errorJava("执行超时,已强制中断");
log.warn("JavaScript执行超时,已强制取消");
} }
}, EXECUTION_TIMEOUT_SECONDS, TimeUnit.SECONDS); }, INDEPENDENT_EXECUTOR);
} catch (java.util.concurrent.RejectedExecutionException e) {
log.warn("演练场线程池已满,任务被拒绝");
close(); // 释放已创建的 ScriptEngine 和 HttpClient 资源
promise.fail(new RuntimeException("演练场线程池已满,请稍后重试", e));
return promise.future();
}
ScheduledFuture<?> timeoutTask = scheduleTimeout(executionFuture, "parse");
// 处理执行结果 // 处理执行结果
executionFuture.whenComplete((result, error) -> { executionFuture.whenComplete((result, error) -> {
@@ -185,7 +225,7 @@ public class JsPlaygroundExecutor {
if (error != null) { if (error != null) {
if (error instanceof CancellationException) { if (error instanceof CancellationException) {
String timeoutMsg = "JavaScript执行超时(超过" + EXECUTION_TIMEOUT_SECONDS + "秒),已强制中断"; String timeoutMsg = "JavaScript执行超时(超过" + EXECUTION_TIMEOUT_SECONDS + "秒),已返回超时并停止外部HTTP资源;ScriptEngine将在执行线程退出后清理";
playgroundLogger.errorJava(timeoutMsg); playgroundLogger.errorJava(timeoutMsg);
log.error(timeoutMsg); log.error(timeoutMsg);
promise.fail(new RuntimeException(timeoutMsg)); promise.fail(new RuntimeException(timeoutMsg));
@@ -210,8 +250,13 @@ public class JsPlaygroundExecutor {
public Future<List<FileInfo>> executeParseFileListAsync() { public Future<List<FileInfo>> executeParseFileListAsync() {
Promise<List<FileInfo>> promise = Promise.promise(); Promise<List<FileInfo>> promise = Promise.promise();
final CompletableFuture<List<FileInfo>> executionFuture;
try {
// 使用独立的ExecutorService执行,避免Vert.x的BlockedThreadChecker输出警告 // 使用独立的ExecutorService执行,避免Vert.x的BlockedThreadChecker输出警告
CompletableFuture<List<FileInfo>> executionFuture = CompletableFuture.supplyAsync(() -> { executionFuture = CompletableFuture.supplyAsync(() -> {
beginExecution();
try {
ScriptEngine engine = engine();
playgroundLogger.infoJava("开始执行parseFileList方法"); playgroundLogger.infoJava("开始执行parseFileList方法");
try { try {
Object parseFileListFunction = engine.get("parseFileList"); Object parseFileListFunction = engine.get("parseFileList");
@@ -242,16 +287,18 @@ public class JsPlaygroundExecutor {
playgroundLogger.errorJava("执行parseFileList方法失败: " + e.getMessage(), e); playgroundLogger.errorJava("执行parseFileList方法失败: " + e.getMessage(), e);
throw new RuntimeException(e); throw new RuntimeException(e);
} }
}, INDEPENDENT_EXECUTOR); } finally {
finishExecution();
// 创建超时任务,强制取消执行
ScheduledFuture<?> timeoutTask = TIMEOUT_SCHEDULER.schedule(() -> {
if (!executionFuture.isDone()) {
executionFuture.cancel(true); // 强制中断执行线程
playgroundLogger.errorJava("执行超时,已强制中断");
log.warn("JavaScript执行超时,已强制取消");
} }
}, EXECUTION_TIMEOUT_SECONDS, TimeUnit.SECONDS); }, INDEPENDENT_EXECUTOR);
} catch (java.util.concurrent.RejectedExecutionException e) {
log.warn("演练场线程池已满,任务被拒绝");
close(); // 释放已创建的 ScriptEngine 和 HttpClient 资源
promise.fail(new RuntimeException("演练场线程池已满,请稍后重试", e));
return promise.future();
}
ScheduledFuture<?> timeoutTask = scheduleTimeout(executionFuture, "parseFileList");
// 处理执行结果 // 处理执行结果
executionFuture.whenComplete((result, error) -> { executionFuture.whenComplete((result, error) -> {
@@ -260,7 +307,7 @@ public class JsPlaygroundExecutor {
if (error != null) { if (error != null) {
if (error instanceof CancellationException) { if (error instanceof CancellationException) {
String timeoutMsg = "JavaScript执行超时(超过" + EXECUTION_TIMEOUT_SECONDS + "秒),已强制中断"; String timeoutMsg = "JavaScript执行超时(超过" + EXECUTION_TIMEOUT_SECONDS + "秒),已返回超时并停止外部HTTP资源;ScriptEngine将在执行线程退出后清理";
playgroundLogger.errorJava(timeoutMsg); playgroundLogger.errorJava(timeoutMsg);
log.error(timeoutMsg); log.error(timeoutMsg);
promise.fail(new RuntimeException(timeoutMsg)); promise.fail(new RuntimeException(timeoutMsg));
@@ -285,8 +332,13 @@ public class JsPlaygroundExecutor {
public Future<String> executeParseByIdAsync() { public Future<String> executeParseByIdAsync() {
Promise<String> promise = Promise.promise(); Promise<String> promise = Promise.promise();
final CompletableFuture<String> executionFuture;
try {
// 使用独立的ExecutorService执行,避免Vert.x的BlockedThreadChecker输出警告 // 使用独立的ExecutorService执行,避免Vert.x的BlockedThreadChecker输出警告
CompletableFuture<String> executionFuture = CompletableFuture.supplyAsync(() -> { executionFuture = CompletableFuture.supplyAsync(() -> {
beginExecution();
try {
ScriptEngine engine = engine();
playgroundLogger.infoJava("开始执行parseById方法"); playgroundLogger.infoJava("开始执行parseById方法");
try { try {
Object parseByIdFunction = engine.get("parseById"); Object parseByIdFunction = engine.get("parseById");
@@ -300,8 +352,9 @@ public class JsPlaygroundExecutor {
Object result = parseByIdMirror.call(null, shareLinkInfoWrapper, httpClient, playgroundLogger); Object result = parseByIdMirror.call(null, shareLinkInfoWrapper, httpClient, playgroundLogger);
if (result instanceof String) { if (result instanceof String) {
playgroundLogger.infoJava("按ID解析成功: " + result); String resultText = limitResultString((String) result, "parseById");
return (String) result; playgroundLogger.infoJava("按ID解析成功,返回结果长度: " + resultText.length());
return resultText;
} else { } else {
String errorMsg = "parseById方法返回值类型错误,期望String,实际: " + String errorMsg = "parseById方法返回值类型错误,期望String,实际: " +
(result != null ? result.getClass().getSimpleName() : "null"); (result != null ? result.getClass().getSimpleName() : "null");
@@ -316,16 +369,18 @@ public class JsPlaygroundExecutor {
playgroundLogger.errorJava("执行parseById方法失败: " + e.getMessage(), e); playgroundLogger.errorJava("执行parseById方法失败: " + e.getMessage(), e);
throw new RuntimeException(e); throw new RuntimeException(e);
} }
}, INDEPENDENT_EXECUTOR); } finally {
finishExecution();
// 创建超时任务,强制取消执行
ScheduledFuture<?> timeoutTask = TIMEOUT_SCHEDULER.schedule(() -> {
if (!executionFuture.isDone()) {
executionFuture.cancel(true); // 强制中断执行线程
playgroundLogger.errorJava("执行超时,已强制中断");
log.warn("JavaScript执行超时,已强制取消");
} }
}, EXECUTION_TIMEOUT_SECONDS, TimeUnit.SECONDS); }, INDEPENDENT_EXECUTOR);
} catch (java.util.concurrent.RejectedExecutionException e) {
log.warn("演练场线程池已满,任务被拒绝");
close(); // 释放已创建的 ScriptEngine 和 HttpClient 资源
promise.fail(new RuntimeException("演练场线程池已满,请稍后重试", e));
return promise.future();
}
ScheduledFuture<?> timeoutTask = scheduleTimeout(executionFuture, "parseById");
// 处理执行结果 // 处理执行结果
executionFuture.whenComplete((result, error) -> { executionFuture.whenComplete((result, error) -> {
@@ -334,7 +389,7 @@ public class JsPlaygroundExecutor {
if (error != null) { if (error != null) {
if (error instanceof CancellationException) { if (error instanceof CancellationException) {
String timeoutMsg = "JavaScript执行超时(超过" + EXECUTION_TIMEOUT_SECONDS + "秒),已强制中断"; String timeoutMsg = "JavaScript执行超时(超过" + EXECUTION_TIMEOUT_SECONDS + "秒),已返回超时并停止外部HTTP资源;ScriptEngine将在执行线程退出后清理";
playgroundLogger.errorJava(timeoutMsg); playgroundLogger.errorJava(timeoutMsg);
log.error(timeoutMsg); log.error(timeoutMsg);
promise.fail(new RuntimeException(timeoutMsg)); promise.fail(new RuntimeException(timeoutMsg));
@@ -355,7 +410,7 @@ public class JsPlaygroundExecutor {
*/ */
public List<JsPlaygroundLogger.LogEntry> getLogs() { public List<JsPlaygroundLogger.LogEntry> getLogs() {
List<JsPlaygroundLogger.LogEntry> logs = playgroundLogger.getLogs(); List<JsPlaygroundLogger.LogEntry> logs = playgroundLogger.getLogs();
System.out.println("[JsPlaygroundExecutor] 获取日志,数量: " + logs.size()); log.debug("获取日志,数量: {}", logs.size());
return logs; return logs;
} }
@@ -373,6 +428,9 @@ public class JsPlaygroundExecutor {
List<FileInfo> fileList = new ArrayList<>(); List<FileInfo> fileList = new ArrayList<>();
if (resultMirror.isArray()) { if (resultMirror.isArray()) {
if (resultMirror.size() > MAX_FILE_LIST_SIZE) {
throw new RuntimeException("文件列表数量超过限制: " + resultMirror.size());
}
for (int i = 0; i < resultMirror.size(); i++) { for (int i = 0; i < resultMirror.size(); i++) {
Object item = resultMirror.get(String.valueOf(i)); Object item = resultMirror.get(String.valueOf(i));
if (item instanceof ScriptObjectMirror) { if (item instanceof ScriptObjectMirror) {
@@ -396,13 +454,13 @@ public class JsPlaygroundExecutor {
// 设置基本字段 // 设置基本字段
if (itemMirror.hasMember("fileName")) { if (itemMirror.hasMember("fileName")) {
fileInfo.setFileName(itemMirror.getMember("fileName").toString()); fileInfo.setFileName(limitField(itemMirror.getMember("fileName")));
} }
if (itemMirror.hasMember("fileId")) { if (itemMirror.hasMember("fileId")) {
fileInfo.setFileId(itemMirror.getMember("fileId").toString()); fileInfo.setFileId(limitField(itemMirror.getMember("fileId")));
} }
if (itemMirror.hasMember("fileType")) { if (itemMirror.hasMember("fileType")) {
fileInfo.setFileType(itemMirror.getMember("fileType").toString()); fileInfo.setFileType(limitField(itemMirror.getMember("fileType")));
} }
if (itemMirror.hasMember("size")) { if (itemMirror.hasMember("size")) {
Object size = itemMirror.getMember("size"); Object size = itemMirror.getMember("size");
@@ -411,16 +469,16 @@ public class JsPlaygroundExecutor {
} }
} }
if (itemMirror.hasMember("sizeStr")) { if (itemMirror.hasMember("sizeStr")) {
fileInfo.setSizeStr(itemMirror.getMember("sizeStr").toString()); fileInfo.setSizeStr(limitField(itemMirror.getMember("sizeStr")));
} }
if (itemMirror.hasMember("createTime")) { if (itemMirror.hasMember("createTime")) {
fileInfo.setCreateTime(itemMirror.getMember("createTime").toString()); fileInfo.setCreateTime(limitField(itemMirror.getMember("createTime")));
} }
if (itemMirror.hasMember("updateTime")) { if (itemMirror.hasMember("updateTime")) {
fileInfo.setUpdateTime(itemMirror.getMember("updateTime").toString()); fileInfo.setUpdateTime(limitField(itemMirror.getMember("updateTime")));
} }
if (itemMirror.hasMember("createBy")) { if (itemMirror.hasMember("createBy")) {
fileInfo.setCreateBy(itemMirror.getMember("createBy").toString()); fileInfo.setCreateBy(limitField(itemMirror.getMember("createBy")));
} }
if (itemMirror.hasMember("downloadCount")) { if (itemMirror.hasMember("downloadCount")) {
Object downloadCount = itemMirror.getMember("downloadCount"); Object downloadCount = itemMirror.getMember("downloadCount");
@@ -429,16 +487,16 @@ public class JsPlaygroundExecutor {
} }
} }
if (itemMirror.hasMember("fileIcon")) { if (itemMirror.hasMember("fileIcon")) {
fileInfo.setFileIcon(itemMirror.getMember("fileIcon").toString()); fileInfo.setFileIcon(limitField(itemMirror.getMember("fileIcon")));
} }
if (itemMirror.hasMember("panType")) { if (itemMirror.hasMember("panType")) {
fileInfo.setPanType(itemMirror.getMember("panType").toString()); fileInfo.setPanType(limitField(itemMirror.getMember("panType")));
} }
if (itemMirror.hasMember("parserUrl")) { if (itemMirror.hasMember("parserUrl")) {
fileInfo.setParserUrl(itemMirror.getMember("parserUrl").toString()); fileInfo.setParserUrl(limitField(itemMirror.getMember("parserUrl")));
} }
if (itemMirror.hasMember("previewUrl")) { if (itemMirror.hasMember("previewUrl")) {
fileInfo.setPreviewUrl(itemMirror.getMember("previewUrl").toString()); fileInfo.setPreviewUrl(limitField(itemMirror.getMember("previewUrl")));
} }
return fileInfo; return fileInfo;
@@ -448,5 +506,135 @@ public class JsPlaygroundExecutor {
return null; return null;
} }
} }
private static String limitResultString(String value, String operation) {
if (value.length() > MAX_RESULT_STRING_LENGTH) {
throw new RuntimeException(operation + " 返回结果过大: " + value.length() + " 字符");
}
return value;
}
private static String limitField(Object value) {
if (value == null) {
return null;
}
String text = value.toString();
if (text.length() > MAX_FILE_FIELD_LENGTH) {
throw new RuntimeException("文件字段过长: " + text.length() + " 字符");
}
return text;
}
private void beginExecution() {
synchronized (lifecycleLock) {
if (closed) {
throw new CancellationException("演练场执行器已关闭");
}
if (running) {
throw new IllegalStateException("演练场执行器已在运行");
}
running = true;
}
}
private ScriptEngine engine() {
ScriptEngine current = engine;
if (current != null) {
return current;
}
synchronized (engineLock) {
if (closed) {
throw new CancellationException("演练场执行器已关闭");
}
if (engine == null) {
engine = initEngine();
}
return engine;
}
}
private void finishExecution() {
synchronized (lifecycleLock) {
running = false;
if (closeRequested) {
doClose();
}
}
}
private ScheduledFuture<?> scheduleTimeout(CompletableFuture<?> executionFuture, String operation) {
// cancel(true) 只能请求中断,Nashorn 死循环不保证立即停止。
return TIMEOUT_SCHEDULER.schedule(() -> {
if (!executionFuture.isDone()) {
executionFuture.cancel(true);
playgroundLogger.errorJava(operation + " 执行超时,已请求取消并停止外部HTTP资源");
forceCloseAfterTimeout();
log.warn("JavaScript {} 执行超时,已请求取消;Nashorn长循环可能继续占用线程,ScriptEngine将在执行线程退出后清理", operation);
}
}, EXECUTION_TIMEOUT_SECONDS, TimeUnit.SECONDS);
}
private void forceCloseAfterTimeout() {
synchronized (lifecycleLock) {
closeRequested = true;
if (running || closed) {
closeExternalResources();
} else {
doClose();
}
}
playgroundLogger.trimToLast(TIMEOUT_LOG_RETAIN);
}
/**
* 释放资源(HttpClient 和 ScriptEngine),避免内存泄漏
* 幂等:可安全多次调用
*/
@Override
public void close() {
synchronized (lifecycleLock) {
closeRequested = true;
if (running || closed) {
closeExternalResources();
return;
}
doClose();
}
}
private void doClose() {
if (closed) return;
closed = true;
closeRequested = false;
closeExternalResources();
cleanupEngine();
log.debug("JsPlaygroundExecutor 资源已释放");
}
private void closeExternalResources() {
if (httpClient != null) {
httpClient.close();
}
}
private void cleanupEngine() {
// 清除 ScriptEngine 的所有 bindings,释放 JS 运行时引用
if (engine != null) {
try {
// 清除注入的 Java 对象引用
engine.put("http", null);
engine.put("logger", null);
engine.put("shareLinkInfo", null);
engine.put("JavaFetch", null);
// 清除所有 ENGINE_SCOPE bindings,包括 eval 加载的 JS 函数
var bindings = engine.getBindings(javax.script.ScriptContext.ENGINE_SCOPE);
if (bindings != null) {
bindings.clear();
}
} catch (Exception e) {
log.warn("清理 ScriptEngine bindings 失败: {}", e.getMessage());
}
}
}
} }
@@ -4,6 +4,9 @@ import java.util.ArrayList;
import java.util.Collections; import java.util.Collections;
import java.util.List; import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/** /**
* 演练场日志收集器 * 演练场日志收集器
* 收集JavaScript执行过程中的日志信息 * 收集JavaScript执行过程中的日志信息
@@ -13,7 +16,11 @@ import java.util.List;
*/ */
public class JsPlaygroundLogger { public class JsPlaygroundLogger {
private static final Logger log = LoggerFactory.getLogger(JsPlaygroundLogger.class);
// 使用线程安全的列表 // 使用线程安全的列表
private static final int MAX_LOG_SIZE = 1000;
private static final int MAX_LOG_MESSAGE_LENGTH = 4096;
private final List<LogEntry> logs = Collections.synchronizedList(new ArrayList<>()); private final List<LogEntry> logs = Collections.synchronizedList(new ArrayList<>());
/** /**
@@ -56,7 +63,23 @@ public class JsPlaygroundLogger {
if (obj == null) { if (obj == null) {
return "null"; return "null";
} }
return obj.toString(); String msg = obj.toString();
if (msg.length() <= MAX_LOG_MESSAGE_LENGTH) {
return msg;
}
return msg.substring(0, MAX_LOG_MESSAGE_LENGTH) + "...[truncated]";
}
/**
* 添加日志条目,超过最大容量时移除最早的条目
*/
private void addLog(LogEntry entry) {
synchronized (logs) {
if (logs.size() >= MAX_LOG_SIZE) {
logs.remove(0);
}
logs.add(entry);
}
} }
/** /**
@@ -67,8 +90,8 @@ public class JsPlaygroundLogger {
*/ */
private void log(String level, Object message, String source) { private void log(String level, Object message, String source) {
String msg = toString(message); String msg = toString(message);
logs.add(new LogEntry(level, msg, source)); addLog(new LogEntry(level, msg, source));
System.out.println("[" + source + "PlaygroundLogger] " + level + ": " + msg); log.debug("[{}PlaygroundLogger] {}: {}", source, level, msg);
} }
/** /**
@@ -109,10 +132,10 @@ public class JsPlaygroundLogger {
public void error(Object message, Throwable throwable) { public void error(Object message, Throwable throwable) {
String msg = toString(message); String msg = toString(message);
if (throwable != null) { if (throwable != null) {
msg = msg + ": " + throwable.getMessage(); msg = toString(msg + ": " + throwable.getMessage());
} }
logs.add(new LogEntry("ERROR", msg, "JS")); addLog(new LogEntry("ERROR", msg, "JS"));
System.out.println("[JSPlaygroundLogger] ERROR: " + msg); log.debug("[JSPlaygroundLogger] ERROR: {}", msg);
} }
// ===== 以下是供Java层调用的内部方法 ===== // ===== 以下是供Java层调用的内部方法 =====
@@ -149,12 +172,12 @@ public class JsPlaygroundLogger {
* 错误日志(带异常,供Java层调用) * 错误日志(带异常,供Java层调用)
*/ */
public void errorJava(String message, Throwable throwable) { public void errorJava(String message, Throwable throwable) {
String msg = message; String msg = toString(message);
if (throwable != null) { if (throwable != null) {
msg = msg + ": " + throwable.getMessage(); msg = toString(msg + ": " + throwable.getMessage());
} }
logs.add(new LogEntry("ERROR", msg, "JAVA")); addLog(new LogEntry("ERROR", msg, "JAVA"));
System.out.println("[JAVAPlaygroundLogger] ERROR: " + msg); log.debug("[JAVAPlaygroundLogger] ERROR: {}", msg);
} }
/** /**
@@ -179,4 +202,17 @@ public class JsPlaygroundLogger {
public void clear() { public void clear() {
logs.clear(); logs.clear();
} }
public void trimToLast(int maxEntries) {
if (maxEntries < 0) {
throw new IllegalArgumentException("maxEntries不能小于0");
}
synchronized (logs) {
int removeCount = logs.size() - maxEntries;
if (removeCount <= 0) {
return;
}
logs.subList(0, removeCount).clear();
}
}
} }
@@ -31,6 +31,8 @@ public class JsScriptLoader {
private static final String RESOURCE_PATH = "custom-parsers"; private static final String RESOURCE_PATH = "custom-parsers";
private static final String EXTERNAL_PATH = "./custom-parsers"; private static final String EXTERNAL_PATH = "./custom-parsers";
private static final long MAX_SCRIPT_SIZE_BYTES = 128 * 1024;
private static final int MAX_EXTERNAL_SCRIPT_COUNT = 100;
// 系统属性配置的外部目录路径 // 系统属性配置的外部目录路径
private static final String EXTERNAL_PATH_PROPERTY = "parser.custom-parsers.path"; private static final String EXTERNAL_PATH_PROPERTY = "parser.custom-parsers.path";
@@ -83,13 +85,15 @@ public class JsScriptLoader {
.getResourceAsStream(resourceFile); .getResourceAsStream(resourceFile);
if (inputStream != null) { if (inputStream != null) {
String jsCode = new String(inputStream.readAllBytes(), StandardCharsets.UTF_8); try (inputStream) {
String jsCode = readResourceScript(inputStream, resourceFile);
CustomParserConfig config = JsScriptMetadataParser.parseScript(jsCode); CustomParserConfig config = JsScriptMetadataParser.parseScript(jsCode);
configs.add(config); configs.add(config);
String fileName = resourceFile.substring(resourceFile.lastIndexOf('/') + 1); String fileName = resourceFile.substring(resourceFile.lastIndexOf('/') + 1);
log.debug("从资源目录加载脚本: {}", fileName); log.debug("从资源目录加载脚本: {}", fileName);
} }
}
} catch (Exception e) { } catch (Exception e) {
log.warn("加载资源脚本失败: {}", resourceFile, e); log.warn("加载资源脚本失败: {}", resourceFile, e);
} }
@@ -139,8 +143,8 @@ public class JsScriptLoader {
try { try {
String jarPath = jarUrl.getPath().substring(5, jarUrl.getPath().indexOf("!")); String jarPath = jarUrl.getPath().substring(5, jarUrl.getPath().indexOf("!"));
JarFile jarFile = new JarFile(jarPath);
try (JarFile jarFile = new JarFile(jarPath)) {
Enumeration<JarEntry> entries = jarFile.entries(); Enumeration<JarEntry> entries = jarFile.entries();
while (entries.hasMoreElements()) { while (entries.hasMoreElements()) {
JarEntry entry = entries.nextElement(); JarEntry entry = entries.nextElement();
@@ -152,8 +156,7 @@ public class JsScriptLoader {
resourceFiles.add(entryName); resourceFiles.add(entryName);
} }
} }
}
jarFile.close();
} catch (Exception e) { } catch (Exception e) {
log.debug("解析JAR包资源文件失败", e); log.debug("解析JAR包资源文件失败", e);
} }
@@ -208,8 +211,10 @@ public class JsScriptLoader {
paths.filter(Files::isRegularFile) paths.filter(Files::isRegularFile)
.filter(path -> path.toString().endsWith(".js")) .filter(path -> path.toString().endsWith(".js"))
.filter(path -> !isExcludedFile(path.getFileName().toString())) .filter(path -> !isExcludedFile(path.getFileName().toString()))
.limit(MAX_EXTERNAL_SCRIPT_COUNT)
.forEach(path -> { .forEach(path -> {
try { try {
ensureScriptSize(path);
String jsCode = Files.readString(path, StandardCharsets.UTF_8); String jsCode = Files.readString(path, StandardCharsets.UTF_8);
CustomParserConfig config = JsScriptMetadataParser.parseScript(jsCode); CustomParserConfig config = JsScriptMetadataParser.parseScript(jsCode);
configs.add(config); configs.add(config);
@@ -263,6 +268,7 @@ public class JsScriptLoader {
throw new IllegalArgumentException("文件不存在: " + filePath); throw new IllegalArgumentException("文件不存在: " + filePath);
} }
ensureScriptSize(path);
String jsCode = Files.readString(path, StandardCharsets.UTF_8); String jsCode = Files.readString(path, StandardCharsets.UTF_8);
return JsScriptMetadataParser.parseScript(jsCode); return JsScriptMetadataParser.parseScript(jsCode);
@@ -285,8 +291,10 @@ public class JsScriptLoader {
throw new IllegalArgumentException("资源文件不存在: " + resourcePath); throw new IllegalArgumentException("资源文件不存在: " + resourcePath);
} }
String jsCode = new String(inputStream.readAllBytes(), StandardCharsets.UTF_8); try (inputStream) {
String jsCode = readResourceScript(inputStream, resourcePath);
return JsScriptMetadataParser.parseScript(jsCode); return JsScriptMetadataParser.parseScript(jsCode);
}
} catch (IOException e) { } catch (IOException e) {
throw new RuntimeException("读取资源文件失败: " + resourcePath, e); throw new RuntimeException("读取资源文件失败: " + resourcePath, e);
@@ -347,4 +355,19 @@ public class JsScriptLoader {
fileName.contains(".test.") || fileName.contains(".test.") ||
fileName.contains(".spec."); fileName.contains(".spec.");
} }
private static void ensureScriptSize(Path path) throws IOException {
long size = Files.size(path);
if (size > MAX_SCRIPT_SIZE_BYTES) {
throw new IllegalArgumentException("JavaScript脚本超过128KB限制: " + path.getFileName());
}
}
private static String readResourceScript(InputStream inputStream, String name) throws IOException {
byte[] bytes = inputStream.readNBytes((int) MAX_SCRIPT_SIZE_BYTES + 1);
if (bytes.length > MAX_SCRIPT_SIZE_BYTES) {
throw new IllegalArgumentException("JavaScript资源脚本超过128KB限制: " + name);
}
return new String(bytes, StandardCharsets.UTF_8);
}
} }
@@ -37,6 +37,7 @@ public class Ce4Tool extends PanBase {
try { try {
URL url = new URL(shareLinkInfo.getShareUrl()); URL url = new URL(shareLinkInfo.getShareUrl());
assertPublicHost(url);
String baseUrl = url.getProtocol() + "://" + url.getHost(); String baseUrl = url.getProtocol() + "://" + url.getHost();
// 如果有端口,拼接上端口 // 如果有端口,拼接上端口
if (url.getPort() != -1) { if (url.getPort() != -1) {
@@ -2,6 +2,7 @@ package cn.qaiu.parser.impl;
import cn.qaiu.entity.FileInfo; import cn.qaiu.entity.FileInfo;
import cn.qaiu.entity.ShareLinkInfo; import cn.qaiu.entity.ShareLinkInfo;
import cn.qaiu.parser.IPanTool;
import cn.qaiu.parser.PanBase; import cn.qaiu.parser.PanBase;
import io.vertx.core.Future; import io.vertx.core.Future;
import io.vertx.core.buffer.Buffer; import io.vertx.core.buffer.Buffer;
@@ -42,6 +43,7 @@ public class CeTool extends PanBase {
String pwd = shareLinkInfo.getSharePassword(); String pwd = shareLinkInfo.getSharePassword();
try { try {
URL url = new URL(shareLinkInfo.getShareUrl()); URL url = new URL(shareLinkInfo.getShareUrl());
assertPublicHost(url);
String baseUrl = url.getProtocol() + "://" + url.getHost(); String baseUrl = url.getProtocol() + "://" + url.getHost();
// 如果有端口,拼接上端口 // 如果有端口,拼接上端口
if (url.getPort() != -1) { if (url.getPort() != -1) {
@@ -160,6 +162,7 @@ public class CeTool extends PanBase {
} catch (Exception e) { } catch (Exception e) {
log.debug("v3 share API解析失败: {}", e.getMessage()); log.debug("v3 share API解析失败: {}", e.getMessage());
} }
tryV4ShareApi(baseUrl, key, pwd);
}).onFailure(t -> { }).onFailure(t -> {
log.debug("v3 share API请求失败: {}", t.getMessage()); log.debug("v3 share API请求失败: {}", t.getMessage());
// 请求失败,尝试 v4 或下一个解析器 // 请求失败,尝试 v4 或下一个解析器
@@ -206,7 +209,8 @@ public class CeTool extends PanBase {
*/ */
private void delegateToCe4Tool() { private void delegateToCe4Tool() {
log.debug("检测到Cloudreve 4.x,转发到Ce4Tool处理"); log.debug("检测到Cloudreve 4.x,转发到Ce4Tool处理");
new Ce4Tool(shareLinkInfo).parse().onComplete(promise); Ce4Tool ce4Tool = new Ce4Tool(shareLinkInfo);
IPanTool.closeAfter(ce4Tool, ce4Tool::parse).onComplete(promise);
} }
@@ -1,36 +1,73 @@
package cn.qaiu.parser.impl; package cn.qaiu.parser.impl;
import cn.qaiu.entity.FileInfo;
import cn.qaiu.entity.ShareLinkInfo; import cn.qaiu.entity.ShareLinkInfo;
import cn.qaiu.parser.PanBase; import cn.qaiu.parser.PanBase;
import cn.qaiu.util.RandomStringGenerator; import cn.qaiu.util.CommonUtils;
import cn.qaiu.util.FileSizeConverter;
import io.vertx.core.Future; import io.vertx.core.Future;
import io.vertx.core.Promise;
import io.vertx.core.buffer.Buffer; import io.vertx.core.buffer.Buffer;
import io.vertx.core.json.JsonArray;
import io.vertx.core.json.JsonObject; import io.vertx.core.json.JsonObject;
import io.vertx.ext.web.client.HttpRequest; import io.vertx.ext.web.client.HttpRequest;
import io.vertx.uritemplate.UriTemplate; import io.vertx.uritemplate.UriTemplate;
import java.net.URI;
import java.net.URLDecoder;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.HashMap; import java.util.HashMap;
import java.util.List;
import java.util.Map; import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/** /**
* <a href="https://www.ctfile.com">诚通网盘</a> * <a href="https://www.ctfile.com">诚通网盘</a>
*/ */
public class CtTool extends PanBase { public class CtTool extends PanBase {
private static final String API_URL_PREFIX = "https://webapi.ctfile.com"; private static final String API_URL_PREFIX = "https://webapi.ctfile.com";
private static final String SHARE_FILE_URL_PREFIX = "https://ctfile.com/file/";
private static final String AJAX_ACCEPT = "application/json, text/javascript, */*; q=0.01";
private static final String BROWSER_UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36";
private static final int FILE_LIST_PAGE_SIZE = 200;
private static final int MAX_FILE_LIST_PAGES = 50;
// https://webapi.ctfile.com/getfile.php?path=f&f=55050874-1246660795-6464f6& // https://webapi.ctfile.com/getfile.php?path=f&f=64115194-17569800420720-06c697&
// passcode=7548&token=30wiijxs1fzhb6brw0p9m6&r=0.5885881231735761& // passcode=7609&r=0.6611183001986635&ref=&url=https%3A%2F%2Furl94.ctfile.com%2Ff%2F64115194-17569800420720-06c697%3Fp%3D7609
// ref=&url=https%3A%2F%2F474b.com%2Ff%2F55050874-1246660795-6464f6%3Fp%3D7548
private static final String API1 = API_URL_PREFIX + "/getfile.php?path={path}" + private static final String API1 = API_URL_PREFIX + "/getfile.php?path={path}" +
"&f={shareKey}&passcode={pwd}&token={token}&r={rand}&ref="; "&f={shareKey}&passcode={pwd}&r={rand}&ref=&url={url}";
//https://webapi.ctfile.com/get_file_url.php?uid=55050874&fid=1246660795&folder_id=0& // https://webapi.ctfile.com/get_down_url.php?uid=64115194&fid=17569800420720&
// file_chk=054bc20461f5c63ff82015b9e69fb7fc&mb=1&token=30wiijxs1fzhb6brw0p9m6&app=0& // file_chk=af5c8757a49cbc69a557eb3da59b246c&start_time=1780471868&wait_seconds=0&rd=0.36...
// acheck=1&verifycode=&rd=0.965929071503574 private static final String API2 = API_URL_PREFIX + "/get_down_url.php?" +
private static final String API2 = API_URL_PREFIX + "/get_file_url.php?" + "uid={uid}&fid={fid}&file_chk={file_chk}" +
"uid={uid}&fid={fid}&folder_id=0&file_chk={file_chk}&mb=0&token={token}&app=0&acheck=0&verifycode=" + "&start_time={start_time}&wait_seconds={wait_seconds}&rd={rand}";
"&rd={rand}";
// https://webapi.ctfile.com/getdir.php?path=d&d=64115194-164803691-48508c&
// folder_id=164803691&fk=decb36&passcode=7609&r=0.23...&ref=&url=https://url94.ctfile.com/d/...
private static final String API_GETDIR = API_URL_PREFIX + "/getdir.php?path={path}" +
"&d={shareKey}&folder_id={folder_id}&fk={fk}&passcode={pwd}&r={rand}&ref=&url={url}";
// DataTables参数,用于获取目录文件列表
private static final String FILE_LIST_PARAMS_TEMPLATE = "&sEcho=1&iColumns=4&sColumns=%2C%2C%2C" +
"&iDisplayStart={start}&iDisplayLength={length}" +
"&mDataProp_0=0&sSearch_0=&bRegex_0=false&bSearchable_0=true&bSortable_0=false" +
"&mDataProp_1=1&sSearch_1=&bRegex_1=false&bSearchable_1=true&bSortable_1=true" +
"&mDataProp_2=2&sSearch_2=&bRegex_2=false&bSearchable_2=true&bSortable_2=true" +
"&mDataProp_3=3&sSearch_3=&bRegex_3=false&bSearchable_3=true&bSortable_3=true" +
"&sSearch=&bRegex=false" +
"&iSortCol_0=3&sSortDir_0=desc&iSortingCols=1";
// 文件列表HTML解析正则
private static final Pattern FILE_ID_PATTERN = Pattern.compile("value=[\"']f(\\d+)[\"']");
private static final Pattern FOLDER_ID_PATTERN = Pattern.compile("value=[\"']d(\\d+)[\"']");
private static final Pattern FILE_HREF_PATTERN = Pattern.compile("href=[\"']#/f/([^\"']+)[\"']");
private static final Pattern FILE_NAME_PATTERN = Pattern.compile("<a\\b[^>]*>([^<]+)</a>", Pattern.CASE_INSENSITIVE);
private static final Pattern FILE_ICON_PATTERN = Pattern.compile("alt=[\"']([^\"']+)[\"']");
private static final Pattern SUBDIR_PATTERN = Pattern.compile("load_subdir\\s*\\((\\d+)\\s*,\\s*['\"]([^'\"]+)['\"]\\)");
/** /**
* 子类重写此构造方法不需要添加额外逻辑 * 子类重写此构造方法不需要添加额外逻辑
@@ -51,62 +88,586 @@ public class CtTool extends PanBase {
@Override @Override
public Future<String> parse() { public Future<String> parse() {
final String shareKey = shareLinkInfo.getShareKey(); final String shareKey = shareLinkInfo.getShareKey();
if (shareKey.indexOf('-') == -1) { if (shareKey == null || shareKey.indexOf('-') == -1) {
fail("shareKey格式不正确找不到'-': {}", shareKey); fail("shareKey格式不正确找不到'-': {}", shareKey);
return promise.future(); return promise.future();
} }
String[] split = shareKey.split("-"); String[] split = shareKey.split("-");
String uid = split[0], fid = split[1]; if (split.length < 2 || split[0].isBlank() || split[1].isBlank()) {
String token = RandomStringGenerator.generateRandomString(); fail("shareKey格式不正确: {}", shareKey);
// 获取url path return promise.future();
int i1 = shareLinkInfo.getShareUrl().indexOf("com/"); }
int i2 = shareLinkInfo.getShareUrl().lastIndexOf("/"); String fallbackUid = split[0], fallbackFid = split[1];
String path = shareLinkInfo.getShareUrl().substring(i1 + 4, i2); String path = extractPath(shareLinkInfo.getShareUrl());
HttpRequest<Buffer> bufferHttpRequest1 = clientSession.getAbs(UriTemplate.of(API1)) HttpRequest<Buffer> bufferHttpRequest1 = withCtAjaxHeaders(clientSession.getAbs(UriTemplate.of(API1))
.setTemplateParam("path", path) .setTemplateParam("path", path)
.setTemplateParam("shareKey", shareKey) .setTemplateParam("shareKey", shareKey)
.setTemplateParam("pwd", shareLinkInfo.getSharePassword()) .setTemplateParam("pwd", shareLinkInfo.getSharePassword())
.setTemplateParam("token", token) .setTemplateParam("rand", String.valueOf(Math.random()))
.setTemplateParam("r", Math.random() + ""); .setTemplateParam("url", shareLinkInfo.getShareUrl()), shareLinkInfo.getShareUrl());
bufferHttpRequest1 bufferHttpRequest1
.send().onSuccess(res -> { .send().onSuccess(res -> {
try {
var resJson = asJson(res); var resJson = asJson(res);
if (resJson.containsKey("file")) { if (resJson == null || resJson.isEmpty()) {
var fileJson = resJson.getJsonObject("file"); fail("解析失败, 上游返回空响应或非JSON响应");
if (fileJson.containsKey("file_chk")) { return;
var file_chk = fileJson.getString("file_chk"); }
HttpRequest<Buffer> bufferHttpRequest2 = clientSession.getAbs(UriTemplate.of(API2)) Object fileValue = resJson.getValue("file");
if (!(fileValue instanceof JsonObject)) {
fail("解析失败, 文件信息为空或格式错误, 可能分享已失效: {}", resJson);
return;
}
var fileJson = (JsonObject) fileValue;
String uid = resolveDownloadUid(fileJson, fallbackUid);
String fid = resolveDownloadFid(fileJson, fallbackFid);
String fileChk = fileJson.getString("file_chk");
String startTime = valueToString(fileJson.getValue("start_time"));
String waitSeconds = valueToString(fileJson.getValue("wait_seconds"));
if (uid.isBlank() || fid.isBlank() || fileChk == null || fileChk.isBlank()
|| startTime.isBlank() || waitSeconds.isBlank()) {
fail("解析失败, 下载参数不完整, 可能分享已失效或者分享密码不对: {}", fileJson);
return;
}
// 提取文件信息并存储
FileInfo fileInfo = new FileInfo()
.setFileName(fileJson.getString("file_name"))
.setFileId(fid)
.setSizeStr(fileJson.getString("file_size"))
.setCreateTime(fileJson.getString("file_time"))
.setCreateBy(fileJson.getString("username"))
.setFileType("file")
.setPanType(shareLinkInfo.getType());
shareLinkInfo.getOtherParam().put("fileInfo", fileInfo);
HttpRequest<Buffer> bufferHttpRequest2 = withCtAjaxHeaders(clientSession.getAbs(UriTemplate.of(API2))
.setTemplateParam("uid", uid) .setTemplateParam("uid", uid)
.setTemplateParam("fid", fid) .setTemplateParam("fid", fid)
.setTemplateParam("file_chk", file_chk) .setTemplateParam("file_chk", fileChk)
.setTemplateParam("token", token) .setTemplateParam("start_time", startTime)
.setTemplateParam("rd", Math.random() + ""); .setTemplateParam("wait_seconds", waitSeconds)
.setTemplateParam("rand", String.valueOf(Math.random())), shareLinkInfo.getShareUrl());
bufferHttpRequest2 bufferHttpRequest2
.send().onSuccess(res2 -> { .send().onSuccess(res2 -> handleDownloadUrlResponse(res2))
JsonObject resJson2 = asJson(res2); .onFailure(t -> fail("下载链接请求失败: {}", t.getMessage()));
if (resJson2.containsKey("downurl")) { } catch (Exception e) {
String downloadUrl = resJson2.getString("downurl"); fail("解析失败: {}", e.getMessage());
}
}).onFailure(t -> fail("文件信息请求失败: {}", t.getMessage()));
return promise.future();
}
private void handleDownloadUrlResponse(io.vertx.ext.web.client.HttpResponse<Buffer> res) {
try {
JsonObject resJson = asJson(res);
if (resJson == null || resJson.isEmpty()) {
fail("解析失败, 下载接口返回空响应或非JSON响应");
return;
}
String downloadUrl = resJson.getString("downurl");
if (downloadUrl == null || downloadUrl.isBlank()) {
fail("解析失败, 可能分享已失效: json: {} 字段 {} 不存在", resJson, "downurl");
return;
}
// 存储下载元数据,包括必要的请求头 // 存储下载元数据,包括必要的请求头
Map<String, String> headers = new HashMap<>(); Map<String, String> headers = new HashMap<>();
headers.put("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"); headers.put("User-Agent", BROWSER_UA);
if (shareLinkInfo.getShareUrl() != null && !shareLinkInfo.getShareUrl().isBlank()) {
headers.put("Referer", shareLinkInfo.getShareUrl()); headers.put("Referer", shareLinkInfo.getShareUrl());
}
// 使用新的 completeWithMeta 方法 // 使用新的 completeWithMeta 方法
completeWithMeta(downloadUrl, headers); completeWithMeta(downloadUrl, headers);
} catch (Exception e) {
fail("解析失败, 下载接口响应处理异常: {}", e.getMessage());
}
}
@Override
public Future<List<FileInfo>> parseFileList() {
Promise<List<FileInfo>> listPromise = Promise.promise();
final String shareKey = shareLinkInfo.getShareKey();
final String shareUrl = shareLinkInfo.getShareUrl();
final String pwd = shareLinkInfo.getSharePassword();
// shareKey格式: uid-folder_id-hash (例如 64115194-164803691-48508c)
if (shareKey == null) {
listPromise.fail(baseMsg() + " shareKey为空");
return listPromise.future();
}
String[] split = shareKey.split("-");
if (split.length < 2) {
listPromise.fail(baseMsg() + " shareKey格式不正确: " + shareKey);
return listPromise.future();
}
String path = extractPath(shareUrl);
Object dirId = shareLinkInfo.getOtherParam() == null ? null : shareLinkInfo.getOtherParam().get("dirId");
DirectoryContext directoryContext = resolveDirectoryContext(shareUrl, dirId);
HttpRequest<Buffer> getDirRequest = withCtAjaxHeaders(clientSession.getAbs(UriTemplate.of(API_GETDIR))
.setTemplateParam("path", path)
.setTemplateParam("shareKey", shareKey)
.setTemplateParam("folder_id", directoryContext.folderId)
.setTemplateParam("fk", directoryContext.folderKey)
.setTemplateParam("pwd", pwd != null ? pwd : "")
.setTemplateParam("rand", String.valueOf(Math.random()))
.setTemplateParam("url", shareUrl), shareUrl);
getDirRequest.send().onSuccess(res -> {
try {
var resJson = asJson(res);
if (resJson == null || resJson.isEmpty()) {
failListPromise(listPromise, baseMsg() + " 目录解析失败: 上游返回空响应或非JSON响应");
return;
}
if (!resJson.containsKey("file")) {
failListPromise(listPromise, baseMsg() + " 目录解析失败: " + resJson.encode());
return;
}
Object dirInfoValue = resJson.getValue("file");
if (!(dirInfoValue instanceof JsonObject)) {
failListPromise(listPromise, baseMsg() + " 目录解析失败: file字段格式错误: " + resJson.encode());
return;
}
JsonObject dirInfo = (JsonObject) dirInfoValue;
Object fileListUrlValue = dirInfo.getValue("url");
String fileListRelUrl = fileListUrlValue instanceof String ? ((String) fileListUrlValue).trim() : "";
if (fileListRelUrl.isBlank()) {
failListPromise(listPromise, baseMsg() + " " + buildDirectoryFailureMessage(resJson, dirInfo));
return;
}
fetchFileListPage(toCtApiUrl(fileListRelUrl), 0, 0, new ArrayList<>(), listPromise,
shareLinkInfo.getType(), getDomainName(), shareUrl, pwd);
} catch (Exception e) {
failListPromise(listPromise, baseMsg() + " 目录解析失败: " + e.getMessage());
}
}).onFailure(t -> failListPromise(listPromise, t));
return listPromise.future();
}
private void fetchFileListPage(String fileListBaseUrl, int start, int pageIndex, List<FileInfo> fileList,
Promise<List<FileInfo>> listPromise, String panType, String domainName,
String shareUrl, String pwd) {
try {
if (pageIndex >= MAX_FILE_LIST_PAGES) {
failListPromise(listPromise, baseMsg() + " 文件列表解析失败: 分页超过最大限制 " + MAX_FILE_LIST_PAGES
+ " (start=" + start + ", length=" + FILE_LIST_PAGE_SIZE + ")");
return;
}
String fileListUrl = appendQueryParams(fileListBaseUrl,
buildFileListParams(start, FILE_LIST_PAGE_SIZE) + "&_=" + System.currentTimeMillis());
withCtAjaxHeaders(clientSession.getAbs(fileListUrl), shareUrl)
.send()
.onSuccess(res -> handleFileListPageResponse(fileListBaseUrl, start, pageIndex, fileList,
listPromise, panType, domainName, shareUrl, pwd, res))
.onFailure(t -> failListPromise(listPromise, t));
} catch (Exception e) {
failListPromise(listPromise, baseMsg() + " 文件列表解析失败: " + e.getMessage()
+ " (start=" + start + ", length=" + FILE_LIST_PAGE_SIZE + ")");
}
}
private void handleFileListPageResponse(String fileListBaseUrl, int start, int pageIndex, List<FileInfo> fileList,
Promise<List<FileInfo>> listPromise, String panType, String domainName,
String shareUrl, String pwd, io.vertx.ext.web.client.HttpResponse<Buffer> res) {
try {
var listJson = asJson(res);
if (listJson == null || listJson.isEmpty()) {
failListPromise(listPromise, baseMsg() + " 文件列表解析失败: 上游返回空响应或非JSON响应"
+ " (start=" + start + ", length=" + FILE_LIST_PAGE_SIZE + ")");
return;
}
Object aaDataValue = listJson.getValue("aaData");
if (!(aaDataValue instanceof JsonArray)) {
failListPromise(listPromise, baseMsg() + " 文件列表解析失败: aaData为空: " + listJson.encode());
return;
}
JsonArray aaData = (JsonArray) aaDataValue;
for (int i = 0; i < aaData.size(); i++) {
try {
Object rowValue = aaData.getValue(i);
if (!(rowValue instanceof JsonArray)) {
log.warn("城通文件列表行格式错误: {}", rowValue);
continue;
}
FileInfo fileInfo = parseFileListRow((JsonArray) rowValue, panType,
domainName, shareUrl, pwd);
if (fileInfo != null) {
fileList.add(fileInfo);
}
} catch (Exception e) {
log.warn("解析文件行失败: {}", e.getMessage());
}
}
int nextStart = start + aaData.size();
int total = parseFileListTotal(listJson);
if (isUnexpectedEmptyFileListPage(start, aaData.size(), total)) {
failListPromise(listPromise, baseMsg() + " 文件列表解析失败: 上游返回空分页"
+ " (start=" + start + ", total=" + total + ")");
return;
}
if (shouldFetchNextFileListPage(start, aaData.size(), total)) {
fetchFileListPage(fileListBaseUrl, nextStart, pageIndex + 1, fileList,
listPromise, panType, domainName, shareUrl, pwd);
} else { } else {
fail("解析失败, 可能分享已失效: json: {} 字段 {} 不存在", resJson2, "downurl"); completeListPromise(listPromise, fileList);
}
} catch (Exception e) {
failListPromise(listPromise, baseMsg() + " 文件列表解析失败: " + e.getMessage()
+ " (start=" + start + ", length=" + FILE_LIST_PAGE_SIZE + ")");
}
}
@Override
public Future<String> parseById() {
Object paramValue = shareLinkInfo.getOtherParam().get("paramJson");
if (!(paramValue instanceof JsonObject)) {
Promise<String> parsePromise = Promise.promise();
parsePromise.fail(baseMsg() + " 缺少下载参数paramJson");
return parsePromise.future();
}
JsonObject paramJson = (JsonObject) paramValue;
if (!applyFileParam(shareLinkInfo, paramJson)) {
Promise<String> parsePromise = Promise.promise();
parsePromise.fail(baseMsg() + " 下载参数id为空");
return parsePromise.future();
}
return parse();
}
static boolean applyFileParam(ShareLinkInfo shareLinkInfo, JsonObject paramJson) {
String fileShareKey = paramJson.getString("id");
if (fileShareKey == null || fileShareKey.isBlank()) {
return false;
}
shareLinkInfo.setSharePassword(paramJson.getString("pwd", ""));
shareLinkInfo.setShareKey(fileShareKey);
shareLinkInfo.setShareUrl(SHARE_FILE_URL_PREFIX + fileShareKey);
shareLinkInfo.setStandardUrl(SHARE_FILE_URL_PREFIX + fileShareKey);
return true;
}
static String resolveDownloadUid(JsonObject fileJson, String fallbackUid) {
return firstNonBlank(valueToString(fileJson.getValue("userid")), fallbackUid);
}
static String resolveDownloadFid(JsonObject fileJson, String fallbackFid) {
return firstNonBlank(valueToString(fileJson.getValue("file_id")), fallbackFid);
}
private HttpRequest<Buffer> withCtAjaxHeaders(HttpRequest<Buffer> request, String shareUrl) {
request.putHeader("User-Agent", BROWSER_UA)
.putHeader("Accept", AJAX_ACCEPT)
.putHeader("X-Requested-With", "XMLHttpRequest");
if (shareUrl != null && !shareUrl.isBlank()) {
request.putHeader("Referer", shareUrl);
}
String origin = extractOrigin(shareUrl);
if (!origin.isBlank()) {
request.putHeader("Origin", origin);
}
return request;
}
static FileInfo parseFileListRow(JsonArray row, String panType, String domainName, String shareUrl, String pwd) {
if (row == null || row.size() < 2) {
return null;
}
String checkboxHtml = rowString(row, 0);
String nameCellHtml = rowString(row, 1);
String sizeStr = rowString(row, 2).trim();
String dateStr = rowString(row, 3).trim();
if (nameCellHtml.isBlank()) {
return null;
}
String fileName = matchFirst(FILE_NAME_PATTERN, nameCellHtml);
String fileIcon = matchFirst(FILE_ICON_PATTERN, nameCellHtml);
if (fileName == null || fileName.isBlank()) {
return null;
}
Matcher subdirMatcher = SUBDIR_PATTERN.matcher(nameCellHtml);
boolean hasSubdirCall = subdirMatcher.find();
if (hasSubdirCall || "folder".equalsIgnoreCase(fileIcon)) {
String folderId = hasSubdirCall ? subdirMatcher.group(1) : null;
String folderKey = hasSubdirCall ? subdirMatcher.group(2) : "";
if (folderId == null) {
folderId = matchFirst(FOLDER_ID_PATTERN, checkboxHtml);
}
if (folderId == null || folderId.isBlank()) {
return null;
}
String dirId = folderId + ":" + folderKey;
FileInfo fileInfo = new FileInfo()
.setFileName(fileName.trim())
.setFileId(folderId)
.setSize(0L)
.setSizeStr(sizeStr.isBlank() ? "0B" : sizeStr)
.setFileType("folder")
.setFileIcon(fileIcon)
.setPanType(panType)
.setParserUrl(buildFolderParserUrl(domainName, shareUrl, dirId, pwd));
if (!dateStr.isBlank()) {
fileInfo.setCreateTime(dateStr).setUpdateTime(dateStr);
}
return fileInfo;
}
String fileShareKey = matchFirst(FILE_HREF_PATTERN, nameCellHtml);
if (fileShareKey == null || fileShareKey.isBlank()) {
return null;
}
String fileId = matchFirst(FILE_ID_PATTERN, checkboxHtml);
JsonObject paramJson = new JsonObject()
.put("id", fileShareKey)
.put("fileName", fileName.trim())
.put("pwd", pwd == null ? "" : pwd);
String param = CommonUtils.urlBase64Encode(paramJson.encode());
long sizeBytes = 0;
try {
sizeBytes = sizeStr.isBlank() ? 0 : FileSizeConverter.convertToBytes(sizeStr);
} catch (Exception ignored) {
}
FileInfo fileInfo = new FileInfo()
.setFileName(fileName.trim())
.setFileId(fileId)
.setSizeStr(sizeStr)
.setSize(sizeBytes)
.setFileType(fileIcon != null ? fileIcon : "file")
.setFileIcon(fileIcon)
.setPanType(panType)
.setParserUrl(String.format("%s/v2/redirectUrl/%s/%s",
domainName, panType, param));
if (!dateStr.isBlank()) {
fileInfo.setCreateTime(dateStr).setUpdateTime(dateStr);
}
return fileInfo;
}
private static String buildFolderParserUrl(String domainName, String shareUrl, String dirId, String pwd) {
String url = String.format("%s/v2/getFileList?url=%s&dirId=%s",
domainName, urlEncode(shareUrl), urlEncode(dirId));
if (pwd != null && !pwd.isBlank()) {
url += "&pwd=" + urlEncode(pwd);
}
return url;
}
static String extractQueryParam(String url, String paramName) {
if (url == null || paramName == null) return null;
int qIdx = url.indexOf('?');
if (qIdx < 0) return null;
String query = url.substring(qIdx + 1);
int fragmentIdx = query.indexOf('#');
if (fragmentIdx >= 0) {
query = query.substring(0, fragmentIdx);
}
for (String param : query.split("&")) {
int eqIdx = param.indexOf('=');
if (eqIdx > 0 && urlDecode(param.substring(0, eqIdx)).equals(paramName)) {
return urlDecode(param.substring(eqIdx + 1));
}
}
return null;
}
static String extractPath(String shareUrl) {
if (shareUrl == null) {
return "";
}
int comIdx = shareUrl.indexOf("com/");
if (comIdx < 0) {
return "";
}
int pathStart = comIdx + 4;
int pathEnd = shareUrl.indexOf('/', pathStart);
if (pathEnd < 0) {
pathEnd = shareUrl.indexOf('?', pathStart);
}
if (pathEnd < 0) {
pathEnd = shareUrl.length();
}
return shareUrl.substring(pathStart, pathEnd);
}
static String extractFolderKey(String shareUrl) {
return trimToEmpty(extractQueryParam(shareUrl, "fk"));
}
static DirectoryContext resolveDirectoryContext(String shareUrl, Object dirIdObj) {
String dirId = dirIdObj == null ? "" : urlDecode(String.valueOf(dirIdObj).trim());
if (!dirId.isBlank()) {
String[] split = dirId.split(":", 2);
return new DirectoryContext(trimToDefault(split[0], "undefined"),
split.length > 1 ? trimToEmpty(split[1]) : "");
}
String queryFolderId = firstNonBlank(extractQueryParam(shareUrl, "folder_id"), extractQueryParam(shareUrl, "d"));
String queryFk = extractFolderKey(shareUrl);
if (!queryFolderId.isBlank() || !queryFk.isBlank()) {
return new DirectoryContext(trimToDefault(queryFolderId, "undefined"), queryFk);
}
return new DirectoryContext("undefined", "");
}
static String buildDirectoryFailureMessage(JsonObject resJson, JsonObject dirInfo) {
String code = valueToString(resJson.getValue("code"));
String message = valueToString(dirInfo.getValue("message"));
if (message != null && !message.isBlank()) {
return "目录解析失败: " + message + " (code=" + code + ")";
}
if ("423".equals(code)) {
return "目录解析失败: 需要访问密码或该分享受限 (code=423)";
}
return "目录解析失败: 文件列表URL为空, 上游响应: " + resJson.encode();
}
static String buildFileListParams(int start, int length) {
return FILE_LIST_PARAMS_TEMPLATE
.replace("{start}", String.valueOf(Math.max(0, start)))
.replace("{length}", String.valueOf(Math.max(1, length)));
}
static int parseFileListTotal(JsonObject listJson) {
int displayTotal = parseInteger(listJson.getValue("iTotalDisplayRecords"), -1);
return displayTotal >= 0 ? displayTotal : parseInteger(listJson.getValue("iTotalRecords"), -1);
}
static boolean shouldFetchNextFileListPage(int start, int rowCount, int total) {
if (rowCount <= 0) {
return false;
}
int fetchedThrough = start + rowCount;
return total < 0 ? rowCount >= FILE_LIST_PAGE_SIZE : fetchedThrough < total;
}
static boolean isUnexpectedEmptyFileListPage(int start, int rowCount, int total) {
return total >= 0 && start < total && rowCount <= 0;
}
private static int parseInteger(Object value, int defaultValue) {
if (value instanceof Number) {
return ((Number) value).intValue();
}
if (value == null) {
return defaultValue;
}
try {
return Integer.parseInt(value.toString());
} catch (NumberFormatException e) {
return defaultValue;
}
}
private static void failListPromise(Promise<List<FileInfo>> listPromise, String message) {
if (!listPromise.future().isComplete()) {
listPromise.fail(message);
}
}
private static void failListPromise(Promise<List<FileInfo>> listPromise, Throwable throwable) {
if (!listPromise.future().isComplete()) {
listPromise.fail(throwable);
}
}
private static void completeListPromise(Promise<List<FileInfo>> listPromise, List<FileInfo> fileList) {
if (!listPromise.future().isComplete()) {
listPromise.complete(fileList);
}
}
private static String toCtApiUrl(String url) {
if (url.startsWith("http://") || url.startsWith("https://")) {
return url;
}
return API_URL_PREFIX + url;
}
private static String appendQueryParams(String url, String params) {
String normalizedParams = params != null && params.startsWith("&") ? params.substring(1) : params;
return url + (url.contains("?") ? "&" : "?") + normalizedParams;
}
private static String rowString(JsonArray row, int index) {
if (row == null || index >= row.size()) {
return "";
}
return valueToString(row.getValue(index));
}
private static String valueToString(Object value) {
return value == null ? "" : value.toString();
}
private static String matchFirst(Pattern pattern, String text) {
if (text == null) {
return null;
}
Matcher matcher = pattern.matcher(text);
return matcher.find() ? matcher.group(1) : null;
}
private static String firstNonBlank(String first, String second) {
return !trimToEmpty(first).isBlank() ? trimToEmpty(first) : trimToEmpty(second);
}
private static String trimToDefault(String value, String defaultValue) {
String result = trimToEmpty(value);
return result.isBlank() ? defaultValue : result;
}
private static String trimToEmpty(String value) {
return value == null ? "" : value.trim();
}
private static String urlEncode(String value) {
return URLEncoder.encode(value == null ? "" : value, StandardCharsets.UTF_8);
}
private static String urlDecode(String value) {
if (value == null) {
return "";
}
try {
return URLDecoder.decode(value, StandardCharsets.UTF_8);
} catch (Exception e) {
return value;
}
}
private static String extractOrigin(String shareUrl) {
try {
URI uri = URI.create(shareUrl);
if (uri.getScheme() == null || uri.getHost() == null) {
return "";
}
String origin = uri.getScheme() + "://" + uri.getHost();
return uri.getPort() > 0 ? origin + ":" + uri.getPort() : origin;
} catch (Exception e) {
return "";
}
}
static final class DirectoryContext {
final String folderId;
final String folderKey;
DirectoryContext(String folderId, String folderKey) {
this.folderId = folderId;
this.folderKey = folderKey;
} }
}).onFailure(handleFail(bufferHttpRequest1.queryParams().toString()));
} else {
fail("解析失败, file_chk找不到, 可能分享已失效或者分享密码不对: {}", fileJson);
}
} else {
fail("解析失败, 文件信息为空, 可能分享已失效");
}
}).onFailure(handleFail(bufferHttpRequest1.queryParams().toString()));
return promise.future();
} }
} }
@@ -25,6 +25,10 @@ public class FcTool extends PanBase {
private static final String DOWN_REQUEST_URL = "https://v2.fangcloud.cn/apps/files/download?file_id={fid}" + private static final String DOWN_REQUEST_URL = "https://v2.fangcloud.cn/apps/files/download?file_id={fid}" +
"&scenario=share&unique_name={uname}"; "&scenario=share&unique_name={uname}";
// 静态编译的正则表达式,避免每次调用都重新编译
private static final Pattern REQUEST_TOKEN_PATTERN = Pattern.compile("name=\"requesttoken\"\\s+value=\"([a-zA-Z0-9_+=]+)\"");
private static final Pattern TYPED_ID_PATTERN = Pattern.compile("id=\"typed_id\"\\s+value=\"file_(\\d+)\"");
public FcTool(ShareLinkInfo shareLinkInfo) { public FcTool(ShareLinkInfo shareLinkInfo) {
super(shareLinkInfo); super(shareLinkInfo);
} }
@@ -41,8 +45,7 @@ public class FcTool extends PanBase {
if (StringUtils.isNotEmpty(pwd)) { if (StringUtils.isNotEmpty(pwd)) {
// 获取requesttoken // 获取requesttoken
String html = res.bodyAsString(); String html = res.bodyAsString();
Pattern compile = Pattern.compile("name=\"requesttoken\"\\s+value=\"([a-zA-Z0-9_+=]+)\""); Matcher matcher = REQUEST_TOKEN_PATTERN.matcher(html);
Matcher matcher = compile.matcher(html);
if (!matcher.find()) { if (!matcher.find()) {
fail(SHARE_URL_PREFIX + " 未匹配到加密分享的密码输入页面的requesttoken"); fail(SHARE_URL_PREFIX + " 未匹配到加密分享的密码输入页面的requesttoken");
return; return;
@@ -71,8 +74,7 @@ public class FcTool extends PanBase {
WebClientSession sClient) { WebClientSession sClient) {
// 从HTML中找到文件id // 从HTML中找到文件id
String html = res.bodyAsString(); String html = res.bodyAsString();
Pattern compile = Pattern.compile("id=\"typed_id\"\\s+value=\"file_(\\d+)\""); Matcher matcher = TYPED_ID_PATTERN.matcher(html);
Matcher matcher = compile.matcher(html);
if (!matcher.find()) { if (!matcher.find()) {
fail(SHARE_URL_PREFIX + " 未匹配到文件id(typed_id)"); fail(SHARE_URL_PREFIX + " 未匹配到文件id(typed_id)");
return; return;
@@ -109,9 +109,9 @@ public class FjTool extends PanBase {
// String uuid = UUID.randomUUID().toString().toLowerCase(); // 也可以使用 UUID.randomUUID().toString() // String uuid = UUID.randomUUID().toString().toLowerCase(); // 也可以使用 UUID.randomUUID().toString()
static String token = null; static volatile String token = null;
static String userId = null; static volatile String userId = null;
public static boolean authFlag = true; public static volatile boolean authFlag = true;
public FjTool(ShareLinkInfo shareLinkInfo) { public FjTool(ShareLinkInfo shareLinkInfo) {
super(shareLinkInfo); super(shareLinkInfo);
@@ -169,8 +169,13 @@ public class FjTool extends PanBase {
// 文件Id // 文件Id
JsonObject fileInfo = resJson.getJsonArray("list").getJsonObject(0); JsonObject fileInfo = resJson.getJsonArray("list").getJsonObject(0);
JsonArray fileListArray = fileInfo.getJsonArray("fileList");
if (fileListArray == null || fileListArray.isEmpty()) {
fail(FIRST_REQUEST_URL + " 文件列表为空: " + fileInfo);
return;
}
// 如果是目录返回目录ID // 如果是目录返回目录ID
JsonObject fileList = fileInfo.getJsonArray("fileList").getJsonObject(0); JsonObject fileList = fileListArray.getJsonObject(0);
if (fileList.getInteger("fileType") == 2) { if (fileList.getInteger("fileType") == 2) {
promise.complete(fileList.getInteger("folderId").toString()); promise.complete(fileList.getInteger("folderId").toString());
return; return;
@@ -289,12 +294,14 @@ public class FjTool extends PanBase {
JsonObject json = asJson(res2); JsonObject json = asJson(res2);
if (json.getInteger("code") == 200) { if (json.getInteger("code") == 200) {
token = json.getJsonObject("data").getString("appToken"); token = json.getJsonObject("data").getString("appToken");
header0.set("appToken", token); MultiMap h0 = MultiMap.caseInsensitiveMultiMap();
log.info("登录成功 token: {}", token); h0.addAll(header0);
h0.set("appToken", token);
log.info("登录成功 token: {}...", token != null ? token.substring(0, Math.min(8, token.length())) : "null");
client.postAbs(UriTemplate.of(TOKEN_VERIFY_URL)) client.postAbs(UriTemplate.of(TOKEN_VERIFY_URL))
.setTemplateParam("uuid", uuid) .setTemplateParam("uuid", uuid)
.setTemplateParam("ts", tsEncode2) .setTemplateParam("ts", tsEncode2)
.putHeaders(header0).send().onSuccess(res -> { .putHeaders(h0).send().onSuccess(res -> {
if (asJson(res).getInteger("code") == 200) { if (asJson(res).getInteger("code") == 200) {
if (FjTool.userId == null) { if (FjTool.userId == null) {
FjTool.userId = asJson(res).getJsonObject("map").getString("userId"); FjTool.userId = asJson(res).getJsonObject("map").getString("userId");
@@ -454,7 +461,10 @@ public class FjTool extends PanBase {
// 如果参数里的目录ID不为空,则直接解析目录 // 如果参数里的目录ID不为空,则直接解析目录
String dirId = (String) shareLinkInfo.getOtherParam().get("dirId"); String dirId = (String) shareLinkInfo.getOtherParam().get("dirId");
if (dirId != null && !dirId.isEmpty()) { if (dirId != null && !dirId.isEmpty()) {
uuid = shareLinkInfo.getOtherParam().get("uuid").toString(); Object uuidObj = shareLinkInfo.getOtherParam().get("uuid");
if (uuidObj != null) {
uuid = uuidObj.toString();
}
parserDir(dirId, shareId, promise0); parserDir(dirId, shareId, promise0);
return promise0.future(); return promise0.future();
} }
@@ -495,7 +505,7 @@ public class FjTool extends PanBase {
JsonArray list; JsonArray list;
try { try {
JsonObject jsonObject = asJson(res); JsonObject jsonObject = asJson(res);
System.out.println(jsonObject.encodePrettily()); log.debug("目录列表: {}", jsonObject.encodePrettily());
list = jsonObject.getJsonArray("list"); list = jsonObject.getJsonArray("list");
} catch (Exception e) { } catch (Exception e) {
log.error("解析目录失败: {}", res.bodyAsString()); log.error("解析目录失败: {}", res.bodyAsString());
@@ -576,6 +586,10 @@ public class FjTool extends PanBase {
// 第二次请求 // 第二次请求
JsonObject paramJson = (JsonObject)shareLinkInfo.getOtherParam().get("paramJson"); JsonObject paramJson = (JsonObject)shareLinkInfo.getOtherParam().get("paramJson");
if (paramJson == null) {
promise.fail("缺少 paramJson 参数");
return promise.future();
}
clientNoRedirects.getAbs(UriTemplate.of(SECOND_REQUEST_URL_VIP)) clientNoRedirects.getAbs(UriTemplate.of(SECOND_REQUEST_URL_VIP))
.setTemplateParam("fidEncode", paramJson.getString("fidEncode")) .setTemplateParam("fidEncode", paramJson.getString("fidEncode"))
.setTemplateParam("uuid", paramJson.getString("uuid")) .setTemplateParam("uuid", paramJson.getString("uuid"))
@@ -389,6 +389,10 @@ public class FsTool extends PanBase {
try { try {
JsonObject paramJson = (JsonObject) shareLinkInfo.getOtherParam().get("paramJson"); JsonObject paramJson = (JsonObject) shareLinkInfo.getOtherParam().get("paramJson");
if (paramJson == null) {
parsePromise.fail("缺少 paramJson 参数");
return parsePromise.future();
}
String shareUrl = paramJson.getString("shareUrl"); String shareUrl = paramJson.getString("shareUrl");
String objToken = paramJson.getString("objToken"); String objToken = paramJson.getString("objToken");
String tenant = extractTenant(shareUrl); String tenant = extractTenant(shareUrl);
@@ -444,7 +448,7 @@ public class FsTool extends PanBase {
if (m1.find()) { if (m1.find()) {
try { try {
return URLDecoder.decode(m1.group(1).trim(), StandardCharsets.UTF_8); return URLDecoder.decode(m1.group(1).trim(), StandardCharsets.UTF_8);
} catch (Exception ignored) { } catch (IllegalArgumentException ignored) {
} }
} }
@@ -453,7 +457,7 @@ public class FsTool extends PanBase {
if (m2.find()) { if (m2.find()) {
try { try {
return URLDecoder.decode(m2.group(1).trim(), StandardCharsets.UTF_8); return URLDecoder.decode(m2.group(1).trim(), StandardCharsets.UTF_8);
} catch (Exception ignored) { } catch (IllegalArgumentException ignored) {
} }
} }
@@ -31,10 +31,12 @@ public class GenShortUrl extends PanBase {
private static final String WRAPPER_URL = "https://www.so.com/link?m=ewgUSYiFWXIoTybC3fJH8YoJy8y10iRquo6cazgINwWjTn3HvVJ92TrCJu0PmMUR0RMDfOAucP3wa4G8j64SrhNH9Z0Cr0PEyn9ASuvpkUGmAjjUEGJkO5%2BIDGWVrEkPHsL7UsoKO6%2BlT%2BD6r&ccc="; private static final String WRAPPER_URL = "https://www.so.com/link?m=ewgUSYiFWXIoTybC3fJH8YoJy8y10iRquo6cazgINwWjTn3HvVJ92TrCJu0PmMUR0RMDfOAucP3wa4G8j64SrhNH9Z0Cr0PEyn9ASuvpkUGmAjjUEGJkO5%2BIDGWVrEkPHsL7UsoKO6%2BlT%2BD6r&ccc=";
private static final String MID = "5095144728824883"; // 微博的mid private static final String MID = "5095144728824883"; // 微博的mid
private static final Pattern SHORT_URL_PATTERN = Pattern.compile("(https?)://t.cn/\\w+");
private static final Pattern COMMENT_ID_PATTERN = Pattern.compile("comment_id=\"(\\d+)\"");
private static final MultiMap HEADER = HeadersMultiMap.headers() private static final MultiMap HEADER = HeadersMultiMap.headers()
.add("Content-Type", "application/x-www-form-urlencoded") .add("Content-Type", "application/x-www-form-urlencoded")
.add("Referer", "https://www.weibo.com") .add("Referer", "https://www.weibo.com")
.add("Content-Type", "application/x-www-form-urlencoded")
.add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36"); .add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36");
Cookie cookie = new DefaultCookie("SUB", "_2A25KJE5vDeRhGeRJ6lsR9SjJzDuIHXVpWM-nrDV8PUJbkNAbLVPlkW1NUmJm3GjYtRHBsHdMUKafkdTL_YheMEmu"); Cookie cookie = new DefaultCookie("SUB", "_2A25KJE5vDeRhGeRJ6lsR9SjJzDuIHXVpWM-nrDV8PUJbkNAbLVPlkW1NUmJm3GjYtRHBsHdMUKafkdTL_YheMEmu");
@@ -64,11 +66,12 @@ public class GenShortUrl extends PanBase {
String shortUrl = extractShortUrl(comment); String shortUrl = extractShortUrl(comment);
if (shortUrl != null) { if (shortUrl != null) {
log.info("生成的短链:{}", shortUrl); log.info("生成的短链:{}", shortUrl);
// 先完成 promise,返回短链
promise.complete(shortUrl);
// 异步清理评论(best-effort,不影响结果)
String commentId = extractCommentId(comment); String commentId = extractCommentId(comment);
if (commentId != null) { if (commentId != null) {
deleteComment(commentId); deleteComment(commentId);
} else {
promise.fail("未能提取评论ID");
} }
} else { } else {
promise.fail("未能生成短链"); promise.fail("未能生成短链");
@@ -103,8 +106,7 @@ public class GenShortUrl extends PanBase {
} }
private String extractShortUrl(String comment) { private String extractShortUrl(String comment) {
Pattern pattern = Pattern.compile("(https?)://t.cn/\\w+"); Matcher matcher = SHORT_URL_PATTERN.matcher(comment);
Matcher matcher = pattern.matcher(comment);
if (matcher.find()) { if (matcher.find()) {
return matcher.group(0); return matcher.group(0);
} }
@@ -112,8 +114,7 @@ public class GenShortUrl extends PanBase {
} }
private String extractCommentId(String comment) { private String extractCommentId(String comment) {
Pattern pattern = Pattern.compile("comment_id=\"(\\d+)\""); Matcher matcher = COMMENT_ID_PATTERN.matcher(comment);
Matcher matcher = pattern.matcher(comment);
if (matcher.find()) { if (matcher.find()) {
return matcher.group(1); return matcher.group(1);
} }
@@ -52,4 +52,14 @@ public class IzSelectorTool implements IPanTool {
public Future<String> parseById() { public Future<String> parseById() {
return selectedTool.parseById(); return selectedTool.parseById();
} }
@Override
public ShareLinkInfo getShareLinkInfo() {
return selectedTool.getShareLinkInfo();
}
@Override
public void close() {
IPanTool.closeQuietly(selectedTool);
}
} }
@@ -89,8 +89,8 @@ public class IzTool extends PanBase {
String uuid = UUID.randomUUID().toString().toLowerCase(); // 也可以使用 UUID.randomUUID().toString() String uuid = UUID.randomUUID().toString().toLowerCase(); // 也可以使用 UUID.randomUUID().toString()
public static String token = null; public static volatile String token = null;
public static boolean authFlag = true; public static volatile boolean authFlag = true;
public Future<String> parse() { public Future<String> parse() {
@@ -102,7 +102,7 @@ public class IzTool extends PanBase {
if (shareLinkInfo.getOtherParam().containsKey("auths")) { if (shareLinkInfo.getOtherParam().containsKey("auths")) {
boolean isTempAuth = shareLinkInfo.getOtherParam().containsKey("__TEMP_AUTH_ADDED"); boolean isTempAuth = shareLinkInfo.getOtherParam().containsKey("__TEMP_AUTH_ADDED");
log.info("文件解析检测到认证信息: isTempAuth={}, authFlag={}, token={}", log.info("文件解析检测到认证信息: isTempAuth={}, authFlag={}, token={}",
isTempAuth, authFlag, token != null ? "已登录(" + token.substring(0, Math.min(10, token.length())) + "...)" : "未登录"); isTempAuth, authFlag, token != null ? "已登录(" + token.substring(0, Math.min(8, token.length())) + "...)" : "未登录");
// 如果需要认证但还没有token,先执行登录 // 如果需要认证但还没有token,先执行登录
if ((isTempAuth || authFlag) && token == null) { if ((isTempAuth || authFlag) && token == null) {
@@ -118,7 +118,7 @@ public class IzTool extends PanBase {
// 登录失败,继续使用免登录模式 // 登录失败,继续使用免登录模式
}); });
} else if (token != null) { } else if (token != null) {
log.info("文件解析使用已有token: {}...", token.substring(0, Math.min(10, token.length()))); log.info("文件解析使用已有token: {}...", token.substring(0, Math.min(8, token.length())));
} }
} else { } else {
log.debug("文件解析无认证信息,使用免登录模式"); log.debug("文件解析无认证信息,使用免登录模式");
@@ -247,7 +247,7 @@ public class IzTool extends PanBase {
log.warn("登录失败: {}", failRes.getMessage()); log.warn("登录失败: {}", failRes.getMessage());
fail(failRes.getMessage()); fail(failRes.getMessage());
}).onSuccess(r-> { }).onSuccess(r-> {
httpRequest.setTemplateParam("appToken", header.get("appToken")) httpRequest.setTemplateParam("appToken", token)
.putHeaders(header); .putHeaders(header);
httpRequest.send().onSuccess(this::down).onFailure(handleFail("请求2")); httpRequest.send().onSuccess(this::down).onFailure(handleFail("请求2"));
}); });
@@ -263,12 +263,12 @@ public class IzTool extends PanBase {
log.warn("重新登录失败: {}", failRes.getMessage()); log.warn("重新登录失败: {}", failRes.getMessage());
fail(failRes.getMessage()); fail(failRes.getMessage());
}).onSuccess(r-> { }).onSuccess(r-> {
httpRequest.setTemplateParam("appToken", header.get("appToken")) httpRequest.setTemplateParam("appToken", token)
.putHeaders(header); .putHeaders(header);
httpRequest.send().onSuccess(this::down).onFailure(handleFail("请求2")); httpRequest.send().onSuccess(this::down).onFailure(handleFail("请求2"));
}); });
} else { } else {
httpRequest.setTemplateParam("appToken", header.get("appToken")) httpRequest.setTemplateParam("appToken", token)
.putHeaders(header); .putHeaders(header);
httpRequest.send().onSuccess(this::down).onFailure(handleFail("请求2")); httpRequest.send().onSuccess(this::down).onFailure(handleFail("请求2"));
} }
@@ -311,8 +311,7 @@ public class IzTool extends PanBase {
JsonObject json = asJson(res2); JsonObject json = asJson(res2);
if (json.getInteger("code") == 200) { if (json.getInteger("code") == 200) {
token = json.getJsonObject("data").getString("appToken"); token = json.getJsonObject("data").getString("appToken");
header.set("appToken", token); log.info("登录成功 token: {}...", token != null ? token.substring(0, Math.min(8, token.length())) : "null");
log.info("登录成功 token: {}", token);
promise1.complete(); promise1.complete();
} else { } else {
// 检查是否为临时认证 // 检查是否为临时认证
@@ -446,11 +445,70 @@ public class IzTool extends PanBase {
private void down(HttpResponse<Buffer> res2) { private void down(HttpResponse<Buffer> res2) {
MultiMap headers = res2.headers(); MultiMap headers = res2.headers();
if (!headers.contains("Location") || StringUtils.isBlank(headers.get("Location"))) { String location = headers.get("Location");
fail("找不到下载链接可能服务器已被禁止或者配置的认证信息有误"); if (StringUtils.isBlank(location)) {
fail("{}", buildMissingLocationMessage(res2));
return; return;
} }
promise.complete(headers.get("Location")); promise.complete(location);
}
private String buildMissingLocationMessage(HttpResponse<Buffer> response) {
StringBuilder message = new StringBuilder("未获取到下载重定向地址");
message.append(", HTTP ").append(response.statusCode());
String body = null;
try {
body = asText(response);
} catch (Exception e) {
body = "<响应体读取失败: " + e.getMessage() + ">";
}
if (StringUtils.isNotBlank(body)) {
try {
JsonObject json = new JsonObject(body);
String upstreamMsg = json.getString("msg");
Object code = json.getValue("code");
if (StringUtils.isNotBlank(upstreamMsg)) {
message.append(", 上游返回: ").append(upstreamMsg);
if (code != null) {
message.append(" (code=").append(code).append(")");
}
} else {
message.append(", 响应体: ").append(previewBody(body));
}
} catch (Exception ignored) {
message.append(", 响应体: ").append(previewBody(body));
}
} else {
message.append(", 响应体为空");
}
Object fileName = shareLinkInfo.getOtherParam().get("fileName");
Object fileSize = shareLinkInfo.getOtherParam().get("fileSizeFormat");
if (fileName != null) {
message.append(", 文件: ").append(fileName);
}
if (fileSize != null) {
message.append(", 大小: ").append(fileSize);
}
if (!hasConfiguredAuth()) {
message.append(", 当前为免登录解析,上游可能要求登录、会员或人工处理");
}
return message.toString();
}
private boolean hasConfiguredAuth() {
Object authObj = shareLinkInfo.getOtherParam().get("auths");
if (!(authObj instanceof MultiMap auths)) {
return false;
}
return StringUtils.isNotBlank(auths.get("username")) && StringUtils.isNotBlank(auths.get("password"));
}
private String previewBody(String body) {
int maxLength = 500;
return body.length() <= maxLength ? body : body.substring(0, maxLength) + "...";
} }
// 目录解析 // 目录解析
@@ -463,7 +521,10 @@ public class IzTool extends PanBase {
// 如果参数里的目录ID不为空,则直接解析目录 // 如果参数里的目录ID不为空,则直接解析目录
String dirId = (String) shareLinkInfo.getOtherParam().get("dirId"); String dirId = (String) shareLinkInfo.getOtherParam().get("dirId");
if (dirId != null && !dirId.isEmpty()) { if (dirId != null && !dirId.isEmpty()) {
uuid = shareLinkInfo.getOtherParam().get("uuid").toString(); Object uuidObj = shareLinkInfo.getOtherParam().get("uuid");
if (uuidObj != null) {
uuid = uuidObj.toString();
}
parserDir(dirId, shareId, promise); parserDir(dirId, shareId, promise);
return promise.future(); return promise.future();
} }
@@ -88,8 +88,8 @@ public class IzToolWithAuth extends PanBase {
String uuid = UUID.randomUUID().toString().toLowerCase(); // 也可以使用 UUID.randomUUID().toString() String uuid = UUID.randomUUID().toString().toLowerCase(); // 也可以使用 UUID.randomUUID().toString()
public static String token = null; public static volatile String token = null;
public static boolean authFlag = true; public static volatile boolean authFlag = true;
public Future<String> parse() { public Future<String> parse() {
@@ -216,7 +216,7 @@ public class IzToolWithAuth extends PanBase {
log.warn("登录失败: {}", failRes.getMessage()); log.warn("登录失败: {}", failRes.getMessage());
fail(failRes.getMessage()); fail(failRes.getMessage());
}).onSuccess(r-> { }).onSuccess(r-> {
httpRequest.setTemplateParam("appToken", header.get("appToken")) httpRequest.setTemplateParam("appToken", token)
.putHeaders(header); .putHeaders(header);
httpRequest.send().onSuccess(this::down).onFailure(handleFail("请求2")); httpRequest.send().onSuccess(this::down).onFailure(handleFail("请求2"));
}); });
@@ -232,12 +232,12 @@ public class IzToolWithAuth extends PanBase {
log.warn("重新登录失败: {}", failRes.getMessage()); log.warn("重新登录失败: {}", failRes.getMessage());
fail(failRes.getMessage()); fail(failRes.getMessage());
}).onSuccess(r-> { }).onSuccess(r-> {
httpRequest.setTemplateParam("appToken", header.get("appToken")) httpRequest.setTemplateParam("appToken", token)
.putHeaders(header); .putHeaders(header);
httpRequest.send().onSuccess(this::down).onFailure(handleFail("请求2")); httpRequest.send().onSuccess(this::down).onFailure(handleFail("请求2"));
}); });
} else { } else {
httpRequest.setTemplateParam("appToken", header.get("appToken")) httpRequest.setTemplateParam("appToken", token)
.putHeaders(header); .putHeaders(header);
httpRequest.send().onSuccess(this::down).onFailure(handleFail("请求2")); httpRequest.send().onSuccess(this::down).onFailure(handleFail("请求2"));
} }
@@ -280,8 +280,7 @@ public class IzToolWithAuth extends PanBase {
JsonObject json = asJson(res2); JsonObject json = asJson(res2);
if (json.getInteger("code") == 200) { if (json.getInteger("code") == 200) {
token = json.getJsonObject("data").getString("appToken"); token = json.getJsonObject("data").getString("appToken");
header.set("appToken", token); log.info("登录成功 token: {}...", token != null ? token.substring(0, Math.min(8, token.length())) : "null");
log.info("登录成功 token: {}", token);
promise1.complete(); promise1.complete();
} else { } else {
// 检查是否为临时认证 // 检查是否为临时认证
@@ -415,11 +414,70 @@ public class IzToolWithAuth extends PanBase {
private void down(HttpResponse<Buffer> res2) { private void down(HttpResponse<Buffer> res2) {
MultiMap headers = res2.headers(); MultiMap headers = res2.headers();
if (!headers.contains("Location") || StringUtils.isBlank(headers.get("Location"))) { String location = headers.get("Location");
fail("找不到下载链接可能服务器已被禁止或者配置的认证信息有误"); if (StringUtils.isBlank(location)) {
fail("{}", buildMissingLocationMessage(res2));
return; return;
} }
promise.complete(headers.get("Location")); promise.complete(location);
}
private String buildMissingLocationMessage(HttpResponse<Buffer> response) {
StringBuilder message = new StringBuilder("未获取到下载重定向地址");
message.append(", HTTP ").append(response.statusCode());
String body = null;
try {
body = asText(response);
} catch (Exception e) {
body = "<响应体读取失败: " + e.getMessage() + ">";
}
if (StringUtils.isNotBlank(body)) {
try {
JsonObject json = new JsonObject(body);
String upstreamMsg = json.getString("msg");
Object code = json.getValue("code");
if (StringUtils.isNotBlank(upstreamMsg)) {
message.append(", 上游返回: ").append(upstreamMsg);
if (code != null) {
message.append(" (code=").append(code).append(")");
}
} else {
message.append(", 响应体: ").append(previewBody(body));
}
} catch (Exception ignored) {
message.append(", 响应体: ").append(previewBody(body));
}
} else {
message.append(", 响应体为空");
}
Object fileName = shareLinkInfo.getOtherParam().get("fileName");
Object fileSize = shareLinkInfo.getOtherParam().get("fileSizeFormat");
if (fileName != null) {
message.append(", 文件: ").append(fileName);
}
if (fileSize != null) {
message.append(", 大小: ").append(fileSize);
}
if (!hasConfiguredAuth()) {
message.append(", 当前为免登录解析,上游可能要求登录、会员或人工处理");
}
return message.toString();
}
private boolean hasConfiguredAuth() {
Object authObj = shareLinkInfo.getOtherParam().get("auths");
if (!(authObj instanceof MultiMap auths)) {
return false;
}
return StringUtils.isNotBlank(auths.get("username")) && StringUtils.isNotBlank(auths.get("password"));
}
private String previewBody(String body) {
int maxLength = 500;
return body.length() <= maxLength ? body : body.substring(0, maxLength) + "...";
} }
// 目录解析 // 目录解析
@@ -432,7 +490,8 @@ public class IzToolWithAuth extends PanBase {
// 如果参数里的目录ID不为空则直接解析目录 // 如果参数里的目录ID不为空则直接解析目录
String dirId = (String) shareLinkInfo.getOtherParam().get("dirId"); String dirId = (String) shareLinkInfo.getOtherParam().get("dirId");
if (dirId != null && !dirId.isEmpty()) { if (dirId != null && !dirId.isEmpty()) {
uuid = shareLinkInfo.getOtherParam().get("uuid").toString(); Object uuidObj = shareLinkInfo.getOtherParam().get("uuid");
uuid = uuidObj != null ? uuidObj.toString() : null;
parserDir(dirId, shareId, promise); parserDir(dirId, shareId, promise);
return promise.future(); return promise.future();
} }
@@ -480,7 +539,7 @@ public class IzToolWithAuth extends PanBase {
requestDirList(id, shareId, tsEncode, promise); requestDirList(id, shareId, tsEncode, promise);
}) })
.onSuccess(r -> { .onSuccess(r -> {
log.info("目录解析登录成功,token={}, 使用 VIP 模式", token != null ? token.substring(0, 10) + "..." : "null"); log.info("目录解析登录成功,token={}, 使用 VIP 模式", token != null ? token.substring(0, Math.min(8, token.length())) + "..." : "null");
requestDirList(id, shareId, tsEncode, promise); requestDirList(id, shareId, tsEncode, promise);
}); });
return; return;
@@ -627,7 +686,7 @@ public class IzToolWithAuth extends PanBase {
// 如果有 token使用 VIP 接口 // 如果有 token使用 VIP 接口
if (StringUtils.isNotBlank(appToken)) { if (StringUtils.isNotBlank(appToken)) {
log.debug("parseById 使用 VIP 接口, appToken={}", appToken.substring(0, Math.min(10, appToken.length())) + "..."); log.debug("parseById 使用 VIP 接口, appToken={}", appToken.substring(0, Math.min(8, appToken.length())) + "...");
webClientSession.getAbs(UriTemplate.of(SECOND_REQUEST_URL_VIP)) webClientSession.getAbs(UriTemplate.of(SECOND_REQUEST_URL_VIP))
.putHeaders(header) .putHeaders(header)
.setTemplateParam("fidEncode", paramJson.getString("fidEncode")) .setTemplateParam("fidEncode", paramJson.getString("fidEncode"))
@@ -15,6 +15,7 @@ import java.net.URLEncoder;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Base64; import java.util.Base64;
import java.util.List; import java.util.List;
import java.util.Random;
import java.util.UUID; import java.util.UUID;
/** /**
@@ -24,6 +25,7 @@ public class LeTool extends PanBase {
private static final String API_URL_PREFIX = "https://lecloud.lenovo.com/mshare/api/clouddiskapi/share/public/v1/"; private static final String API_URL_PREFIX = "https://lecloud.lenovo.com/mshare/api/clouddiskapi/share/public/v1/";
private static final String DEFAULT_FILE_TYPE = "file"; private static final String DEFAULT_FILE_TYPE = "file";
private static final int FILE_TYPE_DIRECTORY = 0; // 目录类型 private static final int FILE_TYPE_DIRECTORY = 0; // 目录类型
private static final Random RANDOM = new Random();
private static final MultiMap HEADERS; private static final MultiMap HEADERS;
@@ -100,8 +102,8 @@ public class LeTool extends PanBase {
} }
String fileId = fileInfoJson.getString("fileId"); String fileId = fileInfoJson.getString("fileId");
// 根据文件ID获取跳转链接 // 根据文件ID获取跳转链接随机选择方式失败自动fallback
getDownURL(dataKey, fileId); getDownURLWithFallback(dataKey, fileId);
} }
} else { } else {
fail("{}: {}", resJson.getString("errcode"), resJson.getString("errmsg")); fail("{}: {}", resJson.getString("errcode"), resJson.getString("errmsg"));
@@ -260,8 +262,8 @@ public class LeTool extends PanBase {
String shareId = paramJson.getString("shareId"); String shareId = paramJson.getString("shareId");
String fileId = paramJson.getString("fileId"); String fileId = paramJson.getString("fileId");
// 调用获取下载链接 // 调用获取下载链接随机选择方式失败自动fallback
getDownURLForById(shareId, fileId, parsePromise); getDownURLWithFallbackForById(shareId, fileId, parsePromise);
} catch (Exception e) { } catch (Exception e) {
parsePromise.fail("解析参数失败: " + e.getMessage()); parsePromise.fail("解析参数失败: " + e.getMessage());
@@ -304,14 +306,22 @@ public class LeTool extends PanBase {
}).onFailure(err -> promise.fail(err)); }).onFailure(err -> promise.fail(err));
} }
private void getDownURL(String key, String fileId) { /**
* 通过 packageDownloadWithFileIds 接口获取下载链接
* 需要两步先获取 downloadUrl再请求 302 跳转
*
* @param shareId 分享ID
* @param fileId 文件ID
* @param promise 完成时会写入此 promise
*/
private void getDownURL(String shareId, String fileId, Promise<String> promise) {
String uuid = UUID.randomUUID().toString(); String uuid = UUID.randomUUID().toString();
JsonArray fileIds = JsonArray.of(fileId); JsonArray fileIds = JsonArray.of(fileId);
String apiUrl2 = API_URL_PREFIX + "packageDownloadWithFileIds"; String apiUrl = API_URL_PREFIX + "packageDownloadWithFileIds";
// {"fileIds":[123],"shareId":"xxx","browserId":"uuid"} // {"fileIds":[123],"shareId":"xxx","browserId":"uuid"}
client.postAbs(apiUrl2) client.postAbs(apiUrl)
.putHeaders(HEADERS) .putHeaders(HEADERS)
.sendJsonObject(JsonObject.of("fileIds", fileIds, "shareId", key, "browserId", uuid)) .sendJsonObject(JsonObject.of("fileIds", fileIds, "shareId", shareId, "browserId", uuid))
.onSuccess(res -> { .onSuccess(res -> {
JsonObject resJson = asJson(res); JsonObject resJson = asJson(res);
if (resJson.containsKey("result")) { if (resJson.containsKey("result")) {
@@ -320,20 +330,107 @@ public class LeTool extends PanBase {
// 获取重定向链接跳转链接 // 获取重定向链接跳转链接
String downloadUrl = dataJson.getString("downloadUrl"); String downloadUrl = dataJson.getString("downloadUrl");
if (downloadUrl == null) { if (downloadUrl == null) {
fail("Result JSON数据异常: downloadUrl不存在"); promise.fail("Result JSON数据异常: downloadUrl不存在");
return; return;
} }
// 获取重定向链接跳转链接 // 获取重定向链接跳转链接
clientNoRedirects.getAbs(downloadUrl).send() clientNoRedirects.getAbs(downloadUrl).send()
.onSuccess(res2 -> promise.complete(res2.headers().get("Location"))) .onSuccess(res2 -> promise.complete(res2.headers().get("Location")))
.onFailure(handleFail(downloadUrl)); .onFailure(err -> promise.fail(err));
} else { } else {
fail("{}: {}", resJson.getString("errcode"), resJson.getString("errmsg")); promise.fail(resJson.getString("errcode") + ": " + resJson.getString("errmsg"));
} }
} else { } else {
fail("Result JSON数据异常: result字段不存在"); promise.fail("Result JSON数据异常: result字段不存在");
}
}).onFailure(err -> promise.fail(err));
}
/**
* 通过 directDownload 接口获取下载链接
* 相比 packageDownloadWithFileIds 少一次请求直接返回302
*
* @param shareId 分享ID
* @param fileId 文件ID
* @param promise 完成时会写入此 promise
*/
private void getDownURLDirect(String shareId, String fileId, Promise<String> promise) {
String uuid = UUID.randomUUID().toString();
String apiUrl = API_URL_PREFIX + "directDownload"
+ "?shareId=" + shareId
+ "&fileId=" + fileId
+ "&browserId=" + uuid;
clientNoRedirects.getAbs(apiUrl)
.putHeaders(HEADERS)
.send()
.onSuccess(res -> {
String location = res.headers().get("Location");
if (location != null && !location.isEmpty()) {
promise.complete(location);
} else {
log.warn("directDownload 返回非302响应: shareId={}, fileId={}, statusCode={}", shareId, fileId, res.statusCode());
promise.fail("directDownload 未返回有效的 Location, statusCode=" + res.statusCode());
}
})
.onFailure(err -> {
log.warn("directDownload 请求失败: shareId={}, fileId={}, error={}", shareId, fileId, err.getMessage());
promise.fail(err);
});
}
/**
* 随机选择下载方式并带 fallback用于 parse
* 先随机选择 directDownload packageDownloadWithFileIds失败则尝试另一个
*/
private void getDownURLWithFallback(String shareId, String fileId) {
boolean useDirect = RANDOM.nextBoolean();
log.info("乐云下载方式选择: shareId={}, fileId={}, method={}", shareId, fileId, useDirect ? "directDownload" : "packageDownloadWithFileIds");
Promise<String> fallbackPromise = Promise.promise();
fallbackPromise.future().onSuccess(url -> {
promise.complete(url);
}).onFailure(err -> {
log.warn("乐云第一种下载方式失败,尝试另一种: {}", err.getMessage());
if (useDirect) {
getDownURL(shareId, fileId, promise);
} else {
getDownURLDirect(shareId, fileId, promise);
}
});
if (useDirect) {
getDownURLDirect(shareId, fileId, fallbackPromise);
} else {
getDownURL(shareId, fileId, fallbackPromise);
}
}
/**
* 随机选择下载方式并带 fallback用于 parseById
* 先随机选择 directDownload packageDownloadWithFileIds失败则尝试另一个
*/
private void getDownURLWithFallbackForById(String shareId, String fileId, Promise<String> promise) {
boolean useDirect = RANDOM.nextBoolean();
log.info("乐云下载方式选择(parseById): shareId={}, fileId={}, method={}", shareId, fileId, useDirect ? "directDownload" : "packageDownloadWithFileIds");
Promise<String> fallbackPromise = Promise.promise();
fallbackPromise.future().onSuccess(url -> {
promise.complete(url);
}).onFailure(err -> {
log.warn("乐云第一种下载方式失败,尝试另一种: {}", err.getMessage());
if (useDirect) {
getDownURLForById(shareId, fileId, promise);
} else {
getDownURLDirect(shareId, fileId, promise);
}
});
if (useDirect) {
getDownURLDirect(shareId, fileId, fallbackPromise);
} else {
getDownURLForById(shareId, fileId, fallbackPromise);
} }
}).onFailure(handleFail(apiUrl2));
} }
/** /**
@@ -14,6 +14,9 @@ import io.vertx.ext.web.client.WebClientSession;
import org.openjdk.nashorn.api.scripting.ScriptObjectMirror; import org.openjdk.nashorn.api.scripting.ScriptObjectMirror;
import javax.script.ScriptException; import javax.script.ScriptException;
import java.net.MalformedURLException;
import java.time.LocalDate;
import java.time.LocalDateTime;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.Map;
@@ -30,6 +33,20 @@ public class LzTool extends PanBase {
WebClientSession webClientSession = WebClientSession.create(clientNoRedirects); WebClientSession webClientSession = WebClientSession.create(clientNoRedirects);
public static final String SHARE_URL_PREFIX = "https://w1.lanzn.com/"; public static final String SHARE_URL_PREFIX = "https://w1.lanzn.com/";
// 静态编译的正则表达式避免每次调用都重新编译
private static final Pattern FILE_NAME_PATTERN = Pattern.compile("padding: 56px 0px 20px 0px;\">(.*?)<|filenajax\">(.*?)<");
private static final Pattern FILE_SIZE_PATTERN = Pattern.compile(">文件大小:</span>(.*?)<br>|\"n_filesize\">大小:(.*?)</div>");
private static final Pattern SHARE_USER_PATTERN = Pattern.compile(">分享用户:</span><font>(.*?)</font>|获取<span>(.*?)</span>的文件|\"user-name\">(.*?)</");
private static final Pattern DESCRIPTION_PATTERN = Pattern.compile("(?s)文件描述:</span><br>(.*?)</td>|class=\"n_box_des\">(.*?)</div>");
private static final Pattern FILE_ID_PATTERN = Pattern.compile("\\?f=(.*?)&|fid = (.*?);");
private static final Pattern CREATE_TIME_PATTERN = Pattern.compile(">上传时间:</span>(.*?)<");
private static final Pattern URL_DATE_PATTERN = Pattern.compile("(\\d{4}/\\d{1,2}/\\d{1,2})");
private static final Pattern ARG1_PATTERN = Pattern.compile("var arg1='([^']+)'");
private static final Pattern IFRAME_SRC_PATTERN = Pattern.compile("src=\"(/fn\\?[a-zA-Z\\d_+/=]{16,})\"");
private static final Pattern RELATIVE_TIME_PATTERN = Pattern.compile("^(\\d+|几)\\s*(分钟|小时)前$");
private static final Pattern DATE_PATTERN = Pattern.compile("^(\\d{4})\\s*[-/年]\\s*(\\d{1,2})\\s*[-/月]\\s*(\\d{1,2})\\s*日?$");
private static final Pattern MONTH_DAY_PATTERN = Pattern.compile("^(\\d{1,2})\\s*月\\s*(\\d{1,2})\\s*日?$");
MultiMap headers0 = HeaderUtils.parseHeaders(""" MultiMap headers0 = HeaderUtils.parseHeaders("""
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate Accept-Encoding: gzip, deflate
@@ -61,42 +78,72 @@ public class LzTool extends PanBase {
client.getAbs(sUrl) client.getAbs(sUrl)
.putHeaders(headers0) .putHeaders(headers0)
.send().onSuccess(res -> { .send().onSuccess(res -> {
try {
String html = asText(res); String html = asText(res);
if (html.contains("var arg1='")) { if (hasAcwArg1(html)) {
webClientSession = WebClientSession.create(clientNoRedirects); webClientSession = WebClientSession.create(clientNoRedirects);
setCookie(html, sUrl); if (!setCookie(html, sUrl)) {
fail("蓝奏云反爬 arg1 Cookie 解析失败,页面内容异常");
return;
}
webClientSession.getAbs(sUrl) webClientSession.getAbs(sUrl)
.putHeaders(headers0) .putHeaders(headers0)
.send().onSuccess(res2 -> { .send().onSuccess(res2 -> {
try {
String html2 = asText(res2); String html2 = asText(res2);
doParser(html2, pwd, sUrl); doParser(html2, pwd, sUrl);
}); } catch (Exception e) {
fail("蓝奏云页面响应处理异常: {}", e.getMessage());
}
}).onFailure(handleFail(sUrl));
} else { } else {
doParser(html, pwd, sUrl); doParser(html, pwd, sUrl);
} }
} catch (Exception e) {
fail("蓝奏云页面响应处理异常: {}", e.getMessage());
}
}).onFailure(handleFail(sUrl)); }).onFailure(handleFail(sUrl));
return promise.future(); return promise.future();
} }
private void doParser(String html, String pwd, String sUrl) { private void doParser(String html, String pwd, String sUrl) {
// 检测是否为目录分享链接 ( /s//b/ 路径段或 b0 开头的路径段) if (html == null || html.isBlank()) {
if (sUrl.matches(".*/(s|b)/[^/]+.*") || sUrl.matches(".*/b0[^/]+.*")) { fail("蓝奏云页面响应为空");
return;
}
if (isShareCancelledPage(html)) {
fail("分享已失效或文件已取消分享");
return;
}
// 检测是否为目录分享链接 ( /s//b/ 路径段或 b 开头的路径段)
if (sUrl.matches(".*/(s|b)/[^/]+.*") || sUrl.matches(".*/b[^/]+.*")) {
fail("该链接为蓝奏云目录分享,请使用目录解析接口"); fail("该链接为蓝奏云目录分享,请使用目录解析接口");
return; return;
} }
// 若仍是校验页 (parse()中cookie域名与实际URL不匹配时会出现), 重试一次 // 若仍是校验页 (parse()中cookie域名与实际URL不匹配时会出现), 重试一次
if (html.contains("var arg1='")) { if (hasAcwArg1(html)) {
webClientSession = WebClientSession.create(clientNoRedirects); webClientSession = WebClientSession.create(clientNoRedirects);
setCookie(html, sUrl); if (!setCookie(html, sUrl)) {
fail("蓝奏云反爬 arg1 Cookie 解析失败,页面内容异常");
return;
}
webClientSession.getAbs(sUrl).putHeaders(headers0).send().onSuccess(res -> { webClientSession.getAbs(sUrl).putHeaders(headers0).send().onSuccess(res -> {
try {
String html2 = asText(res); String html2 = asText(res);
if (html2.contains("var arg1='")) { if (isShareCancelledPage(html2)) {
fail("分享已失效或文件已取消分享");
return;
}
if (hasAcwArg1(html2)) {
fail("蓝奏云反爬校验失败,请稍后重试"); fail("蓝奏云反爬校验失败,请稍后重试");
return; return;
} }
doParserInternal(html2, pwd, sUrl); doParserInternal(html2, pwd, sUrl);
} catch (Exception e) {
fail("蓝奏云页面响应处理异常: {}", e.getMessage());
}
}).onFailure(handleFail(sUrl)); }).onFailure(handleFail(sUrl));
return; return;
} }
@@ -104,14 +151,21 @@ public class LzTool extends PanBase {
} }
private void doParserInternal(String html, String pwd, String sUrl) { private void doParserInternal(String html, String pwd, String sUrl) {
if (html == null || html.isBlank()) {
fail("蓝奏云页面响应为空");
return;
}
if (isShareCancelledPage(html)) {
fail("分享已失效或文件已取消分享");
return;
}
try { try {
setFileInfo(html, shareLinkInfo); setFileInfo(html, shareLinkInfo);
} catch (Exception e) { } catch (Exception e) {
e.printStackTrace(); log.error("文件信息解析异常", e);
} }
// 匹配iframe // 匹配iframe
Pattern compile = Pattern.compile("src=\"(/fn\\?[a-zA-Z\\d_+/=]{16,})\""); Matcher matcher = IFRAME_SRC_PATTERN.matcher(html);
Matcher matcher = compile.matcher(html);
// 没有Iframe说明是加密分享, 匹配sign通过密码请求下载页面 // 没有Iframe说明是加密分享, 匹配sign通过密码请求下载页面
if (!matcher.find()) { if (!matcher.find()) {
try { try {
@@ -125,14 +179,29 @@ public class LzTool extends PanBase {
// 没有密码 // 没有密码
String iframePath = matcher.group(1); String iframePath = matcher.group(1);
String absoluteURI = SHARE_URL_PREFIX + iframePath; String absoluteURI = SHARE_URL_PREFIX + iframePath;
webClientSession.getAbs(absoluteURI).putHeaders(headers0).send().onSuccess(res2 -> { // 创建局部副本避免修改实例字段导致累积
MultiMap headersCopy = MultiMap.caseInsensitiveMultiMap().addAll(headers0);
headersCopy.add("Referer", absoluteURI);
webClientSession.getAbs(absoluteURI).putHeaders(headersCopy).send().onSuccess(res2 -> {
try {
String html2 = asText(res2); String html2 = asText(res2);
if (isShareCancelledPage(html2)) {
fail("分享已失效或文件已取消分享");
return;
}
String jsText = getJsText(html2); String jsText = getJsText(html2);
if (jsText == null) { if (jsText == null) {
headers0.add("Referer", absoluteURI); if (!setCookie(html2, absoluteURI)) {
setCookie(html2, absoluteURI); fail("蓝奏云反爬 arg1 Cookie 解析失败,页面内容异常");
return;
}
webClientSession.getAbs(absoluteURI).send().onSuccess(res3 -> { webClientSession.getAbs(absoluteURI).send().onSuccess(res3 -> {
try {
String html3 = asText(res3); String html3 = asText(res3);
if (isShareCancelledPage(html3)) {
fail("分享已失效或文件已取消分享");
return;
}
String jsText3 = getJsText(html3); String jsText3 = getJsText(html3);
if (jsText3 != null) { if (jsText3 != null) {
try { try {
@@ -144,7 +213,10 @@ public class LzTool extends PanBase {
} else { } else {
fail(SHARE_URL_PREFIX + iframePath + " -> " + sUrl + ": 获取失败0, 可能分享已失效"); fail(SHARE_URL_PREFIX + iframePath + " -> " + sUrl + ": 获取失败0, 可能分享已失效");
} }
}); } catch (Exception e) {
fail("蓝奏云 iframe 响应处理异常: {}", e.getMessage());
}
}).onFailure(handleFail(absoluteURI));
} else { } else {
try { try {
ScriptObjectMirror scriptObjectMirror = JsExecUtils.executeDynamicJs(jsText, null); ScriptObjectMirror scriptObjectMirror = JsExecUtils.executeDynamicJs(jsText, null);
@@ -153,18 +225,18 @@ public class LzTool extends PanBase {
fail(e, "js引擎执行失败"); fail(e, "js引擎执行失败");
} }
} }
} catch (Exception e) {
fail("蓝奏云 iframe 响应处理异常: {}", e.getMessage());
}
}).onFailure(handleFail(SHARE_URL_PREFIX)); }).onFailure(handleFail(SHARE_URL_PREFIX));
} }
} }
private void setCookie(String html, String url) { private boolean setCookie(String html, String url) {
int beginIndex = html.indexOf("arg1='") + 6; String arg1 = extractAcwArg1(html);
int endIndex = html.indexOf("';", beginIndex); if (arg1 == null) {
if (beginIndex < 6 || endIndex == -1 || endIndex <= beginIndex) { return false;
fail("蓝奏云反爬 arg1 Cookie 解析失败,页面内容异常");
return;
} }
String arg1 = html.substring(beginIndex, endIndex);
String acw_sc__v2 = AcwScV2Generator.acwScV2Simple(arg1); String acw_sc__v2 = AcwScV2Generator.acwScV2Simple(arg1);
// URL 中动态提取域名 lanzoum.com, lanzoux.com // URL 中动态提取域名 lanzoum.com, lanzoux.com
String domain = ".lanzn.com"; // 默认兜底 String domain = ".lanzn.com"; // 默认兜底
@@ -175,7 +247,7 @@ public class LzTool extends PanBase {
if (firstDot >= 0) { if (firstDot >= 0) {
domain = host.substring(firstDot); // e.g. ".lanzoum.com" domain = host.substring(firstDot); // e.g. ".lanzoum.com"
} }
} catch (Exception ignored) {} } catch (MalformedURLException ignored) {}
// 创建一个 Cookie 并放入 CookieStore // 创建一个 Cookie 并放入 CookieStore
DefaultCookie nettyCookie = new DefaultCookie("acw_sc__v2", acw_sc__v2); DefaultCookie nettyCookie = new DefaultCookie("acw_sc__v2", acw_sc__v2);
nettyCookie.setDomain(domain); nettyCookie.setDomain(domain);
@@ -183,6 +255,7 @@ public class LzTool extends PanBase {
nettyCookie.setSecure(false); nettyCookie.setSecure(false);
nettyCookie.setHttpOnly(false); nettyCookie.setHttpOnly(false);
webClientSession.cookieStore().put(nettyCookie); webClientSession.cookieStore().put(nettyCookie);
return true;
} }
private String getJsByPwd(String pwd, String html, String subText) { private String getJsByPwd(String pwd, String html, String subText) {
@@ -200,6 +273,9 @@ public class LzTool extends PanBase {
} }
private String getJsText(String html) { private String getJsText(String html) {
if (html == null) {
return null;
}
String jsTagStart = "<script type=\"text/javascript\">"; String jsTagStart = "<script type=\"text/javascript\">";
String jsTagEnd = "</script>"; String jsTagEnd = "</script>";
int index = html.lastIndexOf(jsTagStart); int index = html.lastIndexOf(jsTagStart);
@@ -208,23 +284,52 @@ public class LzTool extends PanBase {
} }
int startPos = index + jsTagStart.length(); int startPos = index + jsTagStart.length();
int endPos = html.indexOf(jsTagEnd, startPos); int endPos = html.indexOf(jsTagEnd, startPos);
if (endPos <= startPos) {
return null;
}
return html.substring(startPos, endPos).replaceAll("<!--.*-->", ""); return html.substring(startPos, endPos).replaceAll("<!--.*-->", "");
} }
static String extractAcwArg1(String html) {
if (html == null) {
return null;
}
int beginIndex = html.indexOf("arg1='");
if (beginIndex < 0) {
return null;
}
beginIndex += 6;
int endIndex = html.indexOf("';", beginIndex);
if (endIndex <= beginIndex) {
return null;
}
return html.substring(beginIndex, endIndex);
}
static boolean isShareCancelledPage(String html) {
return html != null
&& ((html.contains("来晚啦") && html.contains("取消分享"))
|| (html.contains("class=\"off\"") && html.contains("取消分享")));
}
private static boolean hasAcwArg1(String html) {
return html != null && html.contains("var arg1='");
}
private void getDownURL(String key, Map<String, ?> obj) { private void getDownURL(String key, Map<String, ?> obj) {
if (obj == null) { if (obj == null) {
fail("需要访问密码"); fail("需要访问密码");
return; return;
} }
Map<?, ?> signMap = (Map<?, ?>)obj.get("data"); Map<?, ?> signMap = (Map<?, ?>)obj.get("data");
String url0 = obj.get("url").toString(); String url0 = String.valueOf(obj.get("url"));
MultiMap map = MultiMap.caseInsensitiveMultiMap(); MultiMap map = MultiMap.caseInsensitiveMultiMap();
signMap.forEach((k, v) -> { signMap.forEach((k, v) -> {
map.add((String) k, v.toString()); map.add((String) k, v.toString());
}); });
MultiMap headers = HeaderUtils.parseHeaders(""" MultiMap headers = HeaderUtils.parseHeaders("""
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6 Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
Cache-Control: no-cache Cache-Control: no-cache
Connection: keep-alive Connection: keep-alive
@@ -260,13 +365,21 @@ public class LzTool extends PanBase {
headers.remove("Referer"); headers.remove("Referer");
webClientSession.getAbs(downUrl).putHeaders(headers).send() webClientSession.getAbs(downUrl).putHeaders(headers).send()
.onSuccess(res3 -> { .onSuccess(res3 -> {
try {
String location = res3.headers().get("Location"); String location = res3.headers().get("Location");
if (location == null) { if (location == null) {
String text = asText(res3); String text = asText(res3);
if (isShareCancelledPage(text)) {
fail(downUrl + " -> 分享已失效或文件已取消分享");
return;
}
// 使用cookie 再请求一次 // 使用cookie 再请求一次
headers.add("Referer", downUrl); headers.add("Referer", downUrl);
int beginIndex = text.indexOf("arg1='") + 6; String arg1 = extractAcwArg1(text);
String arg1 = text.substring(beginIndex, text.indexOf("';", beginIndex)); if (arg1 == null) {
fail(downUrl + " -> 蓝奏云反爬 arg1 Cookie 解析失败,可能分享已失效");
return;
}
String acw_sc__v2 = AcwScV2Generator.acwScV2Simple(arg1); String acw_sc__v2 = AcwScV2Generator.acwScV2Simple(arg1);
// downUrl 中动态提取域名 // downUrl 中动态提取域名
String downDomain = ".lanrar.com"; String downDomain = ".lanrar.com";
@@ -275,7 +388,7 @@ public class LzTool extends PanBase {
String h = du.getHost(); String h = du.getHost();
int dot = h.indexOf('.'); int dot = h.indexOf('.');
if (dot >= 0) downDomain = h.substring(dot); if (dot >= 0) downDomain = h.substring(dot);
} catch (Exception ignored) {} } catch (MalformedURLException ignored) {}
// 创建一个 Cookie 并放入 CookieStore // 创建一个 Cookie 并放入 CookieStore
DefaultCookie nettyCookie = new DefaultCookie("acw_sc__v2", acw_sc__v2); DefaultCookie nettyCookie = new DefaultCookie("acw_sc__v2", acw_sc__v2);
nettyCookie.setDomain(downDomain); nettyCookie.setDomain(downDomain);
@@ -286,16 +399,23 @@ public class LzTool extends PanBase {
webClientSession2.cookieStore().put(nettyCookie); webClientSession2.cookieStore().put(nettyCookie);
webClientSession2.getAbs(downUrl).putHeaders(headers).send() webClientSession2.getAbs(downUrl).putHeaders(headers).send()
.onSuccess(res4 -> { .onSuccess(res4 -> {
try {
String location0 = res4.headers().get("Location"); String location0 = res4.headers().get("Location");
if (location0 == null) { if (location0 == null) {
fail(downUrl + " -> 直链获取失败2, 可能分享已失效"); fail(downUrl + " -> 直链获取失败2, 可能分享已失效");
} else { } else {
setDateAndComplate(location0); setDateAndComplete(location0);
}
} catch (Exception e) {
fail("蓝奏云直链二次响应处理异常: {}", e.getMessage());
} }
}).onFailure(handleFail(downUrl)); }).onFailure(handleFail(downUrl));
return; return;
} }
setDateAndComplate(location); setDateAndComplete(location);
} catch (Exception e) {
fail("蓝奏云直链响应处理异常: {}", e.getMessage());
}
}) })
.onFailure(handleFail(downUrl)); .onFailure(handleFail(downUrl));
} catch (Exception e) { } catch (Exception e) {
@@ -304,12 +424,11 @@ public class LzTool extends PanBase {
}).onFailure(handleFail(url)); }).onFailure(handleFail(url));
} }
private void setDateAndComplate(String location0) { private void setDateAndComplete(String location0) {
// 分享时间 提取url中的时间戳格式lanzoui.com/abc/abc/yyyy/mm/dd/ // 分享时间 提取url中的时间戳格式lanzoui.com/abc/abc/yyyy/mm/dd/
String regex = "(\\d{4}/\\d{1,2}/\\d{1,2})"; Matcher matcher = URL_DATE_PATTERN.matcher(location0);
Matcher matcher = Pattern.compile(regex).matcher(location0);
if (matcher.find()) { if (matcher.find()) {
String dateStr = matcher.group().replace("/", "-"); String dateStr = parseLanzouFileTime(matcher.group());
((FileInfo)shareLinkInfo.getOtherParam().get("fileInfo")).setCreateTime(dateStr); ((FileInfo)shareLinkInfo.getOtherParam().get("fileInfo")).setCreateTime(dateStr);
} }
promise.complete(location0); promise.complete(location0);
@@ -337,26 +456,45 @@ public class LzTool extends PanBase {
String pwd = shareLinkInfo.getSharePassword(); String pwd = shareLinkInfo.getSharePassword();
webClientSession.getAbs(sUrl).send().onSuccess(res -> { webClientSession.getAbs(sUrl).send().onSuccess(res -> {
try {
String html = asText(res); String html = asText(res);
// 检查是否需要 cookie 验证 // 检查是否需要 cookie 验证
if (html.contains("var arg1='")) { if (hasAcwArg1(html)) {
webClientSession = WebClientSession.create(clientNoRedirects); webClientSession = WebClientSession.create(clientNoRedirects);
setCookie(html, sUrl); if (!setCookie(html, sUrl)) {
promise.tryFail(baseMsg() + "蓝奏云反爬 arg1 Cookie 解析失败,页面内容异常");
return;
}
// 重新请求 // 重新请求
webClientSession.getAbs(sUrl).send().onSuccess(res2 -> { webClientSession.getAbs(sUrl).send().onSuccess(res2 -> {
try {
handleFileListParse(asText(res2), pwd, sUrl, promise); handleFileListParse(asText(res2), pwd, sUrl, promise);
}).onFailure(err -> promise.fail(err)); } catch (Exception e) {
promise.tryFail(e);
}
}).onFailure(promise::tryFail);
return; return;
} }
handleFileListParse(html, pwd, sUrl, promise); handleFileListParse(html, pwd, sUrl, promise);
}).onFailure(err -> promise.fail(err)); } catch (Exception e) {
promise.tryFail(e);
}
}).onFailure(promise::tryFail);
return promise.future(); return promise.future();
} }
private void handleFileListParse(String html, String pwd, String sUrl, Promise<List<FileInfo>> promise) { private void handleFileListParse(String html, String pwd, String sUrl, Promise<List<FileInfo>> promise) {
// 检测是否为文件分享链接 (不含 /s//b/ 路径段且不含 b0 开头的路径段) if (html == null || html.isBlank()) {
if (!sUrl.matches(".*/(s|b)/[^/]+.*") && !sUrl.matches(".*/b0[^/]+.*")) { promise.tryFail(baseMsg() + "蓝奏云页面响应为空");
promise.fail(baseMsg() + "该链接为蓝奏云文件分享,请使用文件解析接口"); return;
}
if (isShareCancelledPage(html)) {
promise.tryFail(baseMsg() + "分享已失效或文件已取消分享");
return;
}
// 检测是否为文件分享链接 (不含 /s//b/ 路径段且不含 b 开头的路径段)
if (!sUrl.matches(".*/(s|b)/[^/]+.*") && !sUrl.matches(".*/b[^/]+.*")) {
promise.tryFail(baseMsg() + "该链接为蓝奏云文件分享,请使用文件解析接口");
return; return;
} }
try { try {
@@ -370,28 +508,43 @@ public class LzTool extends PanBase {
String url = SHARE_URL_PREFIX + "filemoreajax.php?file=" + data.get("fid"); String url = SHARE_URL_PREFIX + "filemoreajax.php?file=" + data.get("fid");
webClientSession.postAbs(url).putHeaders(headers).sendForm(map).onSuccess(res2 -> { webClientSession.postAbs(url).putHeaders(headers).sendForm(map).onSuccess(res2 -> {
try {
String resBody = asText(res2); String resBody = asText(res2);
// 再次检查是否需要 cookie 验证 // 再次检查是否需要 cookie 验证
if (resBody.contains("var arg1='")) { if (hasAcwArg1(resBody)) {
setCookie(resBody, url); if (!setCookie(resBody, url)) {
promise.tryFail(baseMsg() + "蓝奏云反爬 arg1 Cookie 解析失败,页面内容异常");
return;
}
// 重新请求 // 重新请求
webClientSession.postAbs(url).putHeaders(headers).sendForm(map).onSuccess(res3 -> { webClientSession.postAbs(url).putHeaders(headers).sendForm(map).onSuccess(res3 -> {
try {
handleFileListResponse(asText(res3), promise); handleFileListResponse(asText(res3), promise);
}).onFailure(err -> promise.fail(err)); } catch (Exception e) {
promise.tryFail(e);
}
}).onFailure(promise::tryFail);
return; return;
} }
handleFileListResponse(resBody, promise); handleFileListResponse(resBody, promise);
}).onFailure(err -> promise.fail(err)); } catch (Exception e) {
promise.tryFail(e);
}
}).onFailure(promise::tryFail);
} catch (ScriptException | NoSuchMethodException | RuntimeException e) { } catch (ScriptException | NoSuchMethodException | RuntimeException e) {
promise.fail(e); promise.tryFail(e);
} }
} }
private void handleFileListResponse(String responseBody, Promise<List<FileInfo>> promise) { private void handleFileListResponse(String responseBody, Promise<List<FileInfo>> promise) {
try { try {
if (responseBody == null || responseBody.isBlank()) {
promise.tryFail(baseMsg() + "蓝奏云文件列表响应为空");
return;
}
JsonObject fileListJson = new JsonObject(responseBody); JsonObject fileListJson = new JsonObject(responseBody);
if (fileListJson.getInteger("zt") != 1) { if (fileListJson.getInteger("zt") != 1) {
promise.fail(baseMsg() + fileListJson.getString("info")); promise.tryFail(baseMsg() + fileListJson.getString("info"));
return; return;
} }
List<FileInfo> list = new ArrayList<>(); List<FileInfo> list = new ArrayList<>();
@@ -422,7 +575,7 @@ public class LzTool extends PanBase {
String param = CommonUtils.urlBase64Encode(paramJson.encode()); String param = CommonUtils.urlBase64Encode(paramJson.encode());
fileInfo.setFileName(fileName) fileInfo.setFileName(fileName)
.setFileId(id) .setFileId(id)
.setCreateTime(fileJson.getString("time")) .setCreateTime(parseLanzouFileTime(fileJson.getString("time")))
.setFileType(fileJson.getString("icon")) .setFileType(fileJson.getString("icon"))
.setSizeStr(fileJson.getString("size")) .setSizeStr(fileJson.getString("size"))
.setSize(sizeNum) .setSize(sizeNum)
@@ -435,10 +588,46 @@ public class LzTool extends PanBase {
}); });
promise.complete(list); promise.complete(list);
} catch (Exception e) { } catch (Exception e) {
promise.fail(e); promise.tryFail(e);
} }
} }
private static String parseLanzouFileTime(String timeText) {
if (timeText == null || timeText.isBlank()) {
return timeText;
}
String normalized = timeText.trim().replaceAll("\\s+", " ");
Matcher matcher = RELATIVE_TIME_PATTERN.matcher(normalized);
if (matcher.matches()) {
int amount = "".equals(matcher.group(1)) ? 1 : Integer.parseInt(matcher.group(1));
String unit = matcher.group(2);
LocalDateTime time = LocalDateTime.now();
if ("小时".equals(unit)) {
time = time.minusHours(amount);
} else {
time = time.minusMinutes(amount);
}
return time.toLocalDate().toString();
}
matcher = DATE_PATTERN.matcher(normalized);
if (matcher.matches()) {
return LocalDate.of(
Integer.parseInt(matcher.group(1)),
Integer.parseInt(matcher.group(2)),
Integer.parseInt(matcher.group(3))
).toString();
}
matcher = MONTH_DAY_PATTERN.matcher(normalized);
if (matcher.matches()) {
return LocalDate.of(
LocalDate.now().getYear(),
Integer.parseInt(matcher.group(1)),
Integer.parseInt(matcher.group(2))
).toString();
}
return normalized;
}
@Override @Override
public Future<String> parseById() { public Future<String> parseById() {
JsonObject paramJson = (JsonObject) shareLinkInfo.getOtherParam().get("paramJson"); JsonObject paramJson = (JsonObject) shareLinkInfo.getOtherParam().get("paramJson");
@@ -454,13 +643,13 @@ public class LzTool extends PanBase {
shareLinkInfo.getOtherParam().put("fileInfo", fileInfo); shareLinkInfo.getOtherParam().put("fileInfo", fileInfo);
try { try {
// 提取文件名 // 提取文件名
String fileName = CommonUtils.extract(html, Pattern.compile("padding: 56px 0px 20px 0px;\">(.*?)<|filenajax\">(.*?)<")); String fileName = CommonUtils.extract(html, FILE_NAME_PATTERN);
String sizeStr = CommonUtils.extract(html, Pattern.compile(">文件大小:</span>(.*?)<br>|\"n_filesize\">大小:(.*?)</div>")); String sizeStr = CommonUtils.extract(html, FILE_SIZE_PATTERN);
String createBy = CommonUtils.extract(html, Pattern.compile(">分享用户:</span><font>(.*?)</font>|获取<span>(.*?)</span>的文件|\"user-name\">(.*?)</")); String createBy = CommonUtils.extract(html, SHARE_USER_PATTERN);
String description = CommonUtils.extract(html, Pattern.compile("(?s)文件描述:</span><br>(.*?)</td>|class=\"n_box_des\">(.*?)</div>")); String description = CommonUtils.extract(html, DESCRIPTION_PATTERN);
// String icon = CommonUtils.extract(html, Pattern.compile("class=\"n_file_icon\" src=\"(.*?)\"")); // String icon = CommonUtils.extract(html, Pattern.compile("class=\"n_file_icon\" src=\"(.*?)\""));
String fileId = CommonUtils.extract(html, Pattern.compile("\\?f=(.*?)&|fid = (.*?);")); String fileId = CommonUtils.extract(html, FILE_ID_PATTERN);
String createTime = CommonUtils.extract(html, Pattern.compile(">上传时间:</span>(.*?)<")); String createTime = CommonUtils.extract(html, CREATE_TIME_PATTERN);
try { try {
fileInfo.setFileName(fileName) fileInfo.setFileName(fileName)
.setCreateBy(createBy) .setCreateBy(createBy)
@@ -468,7 +657,7 @@ public class LzTool extends PanBase {
.setDescription(description) .setDescription(description)
.setFileType("file") .setFileType("file")
.setFileId(fileId) .setFileId(fileId)
.setCreateTime(createTime); .setCreateTime(parseLanzouFileTime(createTime));
if (sizeStr != null && !sizeStr.isBlank()) { if (sizeStr != null && !sizeStr.isBlank()) {
long bytes = FileSizeConverter.convertToBytes(sizeStr); long bytes = FileSizeConverter.convertToBytes(sizeStr);
fileInfo.setSize(bytes).setSizeStr(FileSizeConverter.convertToReadableSize(bytes)); fileInfo.setSize(bytes).setSizeStr(FileSizeConverter.convertToReadableSize(bytes));
@@ -21,6 +21,8 @@ public class MkgsTool extends PanBase {
public static final String API_URL = "https://m.kugou.com/app/i/getSongInfo.php?cmd=playInfo&hash={hash}"; public static final String API_URL = "https://m.kugou.com/app/i/getSongInfo.php?cmd=playInfo&hash={hash}";
private static final Pattern HASH_PATTERN = Pattern.compile("\"hash\"\\s*:\\s*\"([A-F0-9]+)\"");
private static final MultiMap headers = MultiMap.caseInsensitiveMultiMap(); private static final MultiMap headers = MultiMap.caseInsensitiveMultiMap();
static { static {
// 设置 User-Agent // 设置 User-Agent
@@ -78,18 +80,15 @@ public class MkgsTool extends PanBase {
protected void downUrl(String locationURL) { protected void downUrl(String locationURL) {
client.getAbs(locationURL).putHeaders(headers).send().onSuccess(res2->{ client.getAbs(locationURL).putHeaders(headers).send().onSuccess(res2->{
String body = res2.bodyAsString(); String body = res2.bodyAsString();
// 正则表达式匹配 hash 字段 Matcher matcher = HASH_PATTERN.matcher(body);
String regex = "\"hash\"\s*:\s*\"([A-F0-9]+)\"";
Pattern pattern = Pattern.compile(regex);
Matcher matcher = pattern.matcher(body);
// 查找并输出 hash 字段的值 // 查找并输出 hash 字段的值
if (matcher.find()) { if (matcher.find()) {
String hashValue = matcher.group(1); // 获取第一个捕获组 String hashValue = matcher.group(1); // 获取第一个捕获组
System.out.println(hashValue); log.debug("hash: {}", hashValue);
client.getAbs(UriTemplate.of(API_URL)).setTemplateParam("hash", hashValue).send().onSuccess(res3 -> { client.getAbs(UriTemplate.of(API_URL)).setTemplateParam("hash", hashValue).send().onSuccess(res3 -> {
JsonObject jsonObject = asJson(res3); JsonObject jsonObject = asJson(res3);
System.out.println(jsonObject.encodePrettily()); log.debug("API response: {}", jsonObject.encodePrettily());
if (jsonObject.containsKey("url")) { if (jsonObject.containsKey("url")) {
promise.complete(jsonObject.getString("url")); promise.complete(jsonObject.getString("url"));
} else { } else {
@@ -19,6 +19,8 @@ public class MkwTool extends PanBase {
public static final String API_URL = "https://www.kuwo.cn/api/v1/www/music/playUrl?mid={mid}&type=music&httpsStatus=1&reqId=&plat=web_www&from="; public static final String API_URL = "https://www.kuwo.cn/api/v1/www/music/playUrl?mid={mid}&type=music&httpsStatus=1&reqId=&plat=web_www&from=";
private static final Pattern COOKIE_PATTERN = Pattern.compile("([A-Za-z0-9_]+)=([A-Za-z0-9]+)");
public MkwTool(ShareLinkInfo shareLinkInfo) { public MkwTool(ShareLinkInfo shareLinkInfo) {
super(shareLinkInfo); super(shareLinkInfo);
@@ -29,19 +31,24 @@ public class MkwTool extends PanBase {
clientSession.getAbs(shareUrl).send().onSuccess(result -> { clientSession.getAbs(shareUrl).send().onSuccess(result -> {
String cookie = result.headers().get("set-cookie"); String cookie = result.headers().get("set-cookie");
if (!cookie.isEmpty()) { if (cookie == null || cookie.isEmpty()) {
fail("未获取到 cookie,无法继续解析");
return;
}
String regex = "([A-Za-z0-9_]+)=([A-Za-z0-9]+)"; Matcher matcher = COOKIE_PATTERN.matcher(cookie);
Pattern pattern = Pattern.compile(regex); if (!matcher.find()) {
Matcher matcher = pattern.matcher(cookie); fail("cookie 格式不匹配");
if (matcher.find()) { return;
System.out.println(matcher.group(1)); }
System.out.println(matcher.group(2));
log.debug("cookie key: {}", matcher.group(1));
log.debug("cookie value: {}", matcher.group(2));
var key = matcher.group(1); var key = matcher.group(1);
var token = matcher.group(2); var token = matcher.group(2);
String sign = JsExecUtils.getKwSign(token, key); String sign = JsExecUtils.getKwSign(token, key);
System.out.println(sign); log.debug("sign: {}", sign);
clientSession.getAbs(UriTemplate.of(API_URL)).setTemplateParam("mid", shareLinkInfo.getShareKey()) clientSession.getAbs(UriTemplate.of(API_URL)).setTemplateParam("mid", shareLinkInfo.getShareKey())
.putHeader("Secret", sign).send().onSuccess(res -> { .putHeader("Secret", sign).send().onSuccess(res -> {
JsonObject json = asJson(res); JsonObject json = asJson(res);
@@ -54,14 +61,11 @@ public class MkwTool extends PanBase {
} }
} catch (Exception e) { } catch (Exception e) {
e.printStackTrace(); log.error("解析失败", e);
fail("解析失败"); fail("解析失败");
} }
}); }).onFailure(handleFail("获取下载链接失败"));
} }).onFailure(handleFail("请求分享页面失败"));
}
});
return promise.future(); return promise.future();
} }
@@ -21,6 +21,8 @@ public class P115Tool extends PanBase {
private static final String SECOND_REQUEST_URL = API_URL_PREFIX + "share/skip_login_downurl"; private static final String SECOND_REQUEST_URL = API_URL_PREFIX + "share/skip_login_downurl";
private static final String DEFAULT_UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36";
private static final MultiMap header; private static final MultiMap header;
static { static {
@@ -49,9 +51,11 @@ public class P115Tool extends PanBase {
public Future<String> parse() { public Future<String> parse() {
// 第一次请求 获取文件信息 // 第一次请求 获取文件信息
Object uaObj = shareLinkInfo.getOtherParam().get("UA");
String ua = uaObj != null ? uaObj.toString() : DEFAULT_UA;
client.getAbs(UriTemplate.of(FIRST_REQUEST_URL)) client.getAbs(UriTemplate.of(FIRST_REQUEST_URL))
.putHeaders(header) .putHeaders(header)
.putHeader("User-Agent", shareLinkInfo.getOtherParam().get("UA").toString()) .putHeader("User-Agent", ua)
.setTemplateParam("dataKey", shareLinkInfo.getShareKey()) .setTemplateParam("dataKey", shareLinkInfo.getShareKey())
.setTemplateParam("dataPwd", shareLinkInfo.getSharePassword()) .setTemplateParam("dataPwd", shareLinkInfo.getSharePassword())
.send().onSuccess(res -> { .send().onSuccess(res -> {
@@ -68,7 +72,7 @@ public class P115Tool extends PanBase {
// share_code={dataKey}&receive_code={dataPwd}&file_id={file_id} // share_code={dataKey}&receive_code={dataPwd}&file_id={file_id}
client.postAbs(SECOND_REQUEST_URL) client.postAbs(SECOND_REQUEST_URL)
.putHeader("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8") .putHeader("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8")
.putHeader("User-Agent", shareLinkInfo.getOtherParam().get("UA").toString()) .putHeader("User-Agent", ua)
.sendForm(MultiMap.caseInsensitiveMultiMap() .sendForm(MultiMap.caseInsensitiveMultiMap()
.set("share_code", shareLinkInfo.getShareKey()) .set("share_code", shareLinkInfo.getShareKey())
.set("receive_code", shareLinkInfo.getSharePassword()) .set("receive_code", shareLinkInfo.getSharePassword())
@@ -16,6 +16,9 @@ import java.util.regex.Pattern;
* 下载链接需要Referer: https://link.yunpan.com/ * 下载链接需要Referer: https://link.yunpan.com/
*/ */
public class P360Tool extends PanBase { public class P360Tool extends PanBase {
private static final Pattern NID_PATTERN = Pattern.compile("\"nid\": \"([^\"]+)\"");
public P360Tool(ShareLinkInfo shareLinkInfo) { public P360Tool(ShareLinkInfo shareLinkInfo) {
super(shareLinkInfo); super(shareLinkInfo);
} }
@@ -43,9 +46,7 @@ public class P360Tool extends PanBase {
clientSession.getAbs(url) clientSession.getAbs(url)
.send() .send()
.onSuccess(res -> { .onSuccess(res -> {
// find "nid": "17402043311959599" Matcher matcher = NID_PATTERN.matcher(res.bodyAsString());
Pattern compile = Pattern.compile("\"nid\": \"([^\"]+)\"");
Matcher matcher = compile.matcher(res.bodyAsString());
AtomicReference<String> nid = new AtomicReference<>(); AtomicReference<String> nid = new AtomicReference<>();
if (matcher.find()) { if (matcher.find()) {
nid.set(matcher.group(1)); nid.set(matcher.group(1));
@@ -69,7 +70,7 @@ public class P360Tool extends PanBase {
clientSession.getAbs(url) clientSession.getAbs(url)
.send() .send()
.onSuccess(res3 -> { .onSuccess(res3 -> {
Matcher matcher1 = compile.matcher(res3.bodyAsString()); Matcher matcher1 = NID_PATTERN.matcher(res3.bodyAsString());
if (matcher1.find()) { if (matcher1.find()) {
nid.set(matcher1.group(1)); nid.set(matcher1.group(1));
} else { } else {
@@ -14,6 +14,25 @@ import java.util.regex.Pattern;
*/ */
public class PcxTool extends PanBase { public class PcxTool extends PanBase {
private static final Pattern TITLE_PATTERN =
Pattern.compile("<title>([^<]+)</title>");
private static final Pattern FILENAME_INPUT_PATTERN =
Pattern.compile("<input id=\"filename\" type=\"hidden\" value=\"([^\"]+)\"");
private static final Pattern FILESIZE_PATTERN =
Pattern.compile("['\"]filesize['\"]\\s*:\\s*['\"]([^'\"]+)['\"]");
private static final Pattern SUFFIX_PATTERN =
Pattern.compile("['\"]suffix['\"]\\s*:\\s*['\"]([^'\"]+)['\"]");
private static final Pattern OBJECT_ID_PATTERN =
Pattern.compile("['\"]objectId['\"]\\s*:\\s*['\"]([^'\"]+)['\"]");
private static final Pattern CREATOR_PATTERN =
Pattern.compile("['\"]creator['\"]\\s*:\\s*['\"]([^'\"]+)['\"]");
private static final Pattern UPLOAD_DATE_PATTERN =
Pattern.compile("['\"]uploadDate['\"]\\s*:\\s*(\\d+)");
private static final Pattern THUMBNAIL_PATTERN =
Pattern.compile("['\"]thumbnail['\"]\\s*:\\s*['\"]([^'\"]+)['\"]");
private static final Pattern DOWNLOAD_PATTERN =
Pattern.compile("['\"]download['\"]\\s*:\\s*['\"]([^'\"]+)['\"]");
public PcxTool(ShareLinkInfo shareLinkInfo) { public PcxTool(ShareLinkInfo shareLinkInfo) {
super(shareLinkInfo); super(shareLinkInfo);
} }
@@ -44,9 +63,7 @@ public class PcxTool extends PanBase {
* 从HTML中提取download链接 * 从HTML中提取download链接
*/ */
private String extractDownloadUrl(String html) { private String extractDownloadUrl(String html) {
// 匹配 'download': 'https://xxx' "download": "https://xxx" Matcher matcher = DOWNLOAD_PATTERN.matcher(html);
Pattern pattern = Pattern.compile("['\"]download['\"]\\s*:\\s*['\"]([^'\"]+)['\"]");
Matcher matcher = pattern.matcher(html);
if (matcher.find()) { if (matcher.find()) {
return matcher.group(1); return matcher.group(1);
} }
@@ -61,13 +78,13 @@ public class PcxTool extends PanBase {
FileInfo fileInfo = new FileInfo(); FileInfo fileInfo = new FileInfo();
// 提取文件名<title>标签或文件名input // 提取文件名<title>标签或文件名input
String fileName = extractByRegex(html, "<title>([^<]+)</title>"); String fileName = extractByRegex(html, TITLE_PATTERN);
if (fileName == null) { if (fileName == null) {
fileName = extractByRegex(html, "<input id=\"filename\" type=\"hidden\" value=\"([^\"]+)\""); fileName = extractByRegex(html, FILENAME_INPUT_PATTERN);
} }
// 提取文件大小'filesize': 'xxx' "filesize": "xxx" // 提取文件大小'filesize': 'xxx' "filesize": "xxx"
String fileSizeStr = extractByRegex(html, "['\"]filesize['\"]\\s*:\\s*['\"]([^'\"]+)['\"]"); String fileSizeStr = extractByRegex(html, FILESIZE_PATTERN);
Long fileSize = null; Long fileSize = null;
if (fileSizeStr != null) { if (fileSizeStr != null) {
try { try {
@@ -76,19 +93,19 @@ public class PcxTool extends PanBase {
} }
// 提取文件类型/后缀'suffix': 'xxx' "suffix": "xxx" // 提取文件类型/后缀'suffix': 'xxx' "suffix": "xxx"
String suffix = extractByRegex(html, "['\"]suffix['\"]\\s*:\\s*['\"]([^'\"]+)['\"]"); String suffix = extractByRegex(html, SUFFIX_PATTERN);
// 提取objectId文件ID'objectId': 'xxx' "objectId": "xxx" // 提取objectId文件ID'objectId': 'xxx' "objectId": "xxx"
String objectId = extractByRegex(html, "['\"]objectId['\"]\\s*:\\s*['\"]([^'\"]+)['\"]"); String objectId = extractByRegex(html, OBJECT_ID_PATTERN);
// 提取创建者'creator': 'xxx' "creator": "xxx" // 提取创建者'creator': 'xxx' "creator": "xxx"
String creator = extractByRegex(html, "['\"]creator['\"]\\s*:\\s*['\"]([^'\"]+)['\"]"); String creator = extractByRegex(html, CREATOR_PATTERN);
// 提取上传时间'uploadDate': timestamp // 提取上传时间'uploadDate': timestamp
String uploadDate = extractByRegex(html, "['\"]uploadDate['\"]\\s*:\\s*(\\d+)"); String uploadDate = extractByRegex(html, UPLOAD_DATE_PATTERN);
// 提取缩略图'thumbnail': 'xxx' "thumbnail": "xxx" // 提取缩略图'thumbnail': 'xxx' "thumbnail": "xxx"
String thumbnail = extractByRegex(html, "['\"]thumbnail['\"]\\s*:\\s*['\"]([^'\"]+)['\"]"); String thumbnail = extractByRegex(html, THUMBNAIL_PATTERN);
// 设置文件信息 // 设置文件信息
if (fileName != null) { if (fileName != null) {
@@ -141,8 +158,7 @@ public class PcxTool extends PanBase {
/** /**
* 使用正则表达式提取内容 * 使用正则表达式提取内容
*/ */
private String extractByRegex(String text, String regex) { private String extractByRegex(String text, Pattern pattern) {
Pattern pattern = Pattern.compile(regex);
Matcher matcher = pattern.matcher(text); Matcher matcher = pattern.matcher(text);
if (matcher.find()) { if (matcher.find()) {
return matcher.group(1); return matcher.group(1);
@@ -28,6 +28,7 @@ public class PdbTool extends PanBase implements IPanTool {
private static final String API_URL = private static final String API_URL =
"https://www.dropbox.com/sharing/fetch_user_content_link"; "https://www.dropbox.com/sharing/fetch_user_content_link";
static final String COOKIE_KEY = "__Host-js_csrf="; static final String COOKIE_KEY = "__Host-js_csrf=";
private static final Pattern CSRF_TOKEN_PATTERN = Pattern.compile(COOKIE_KEY + "([\\w-]+);");
public PdbTool(ShareLinkInfo shareLinkInfo) { public PdbTool(ShareLinkInfo shareLinkInfo) {
super(shareLinkInfo); super(shareLinkInfo);
@@ -47,7 +48,7 @@ public class PdbTool extends PanBase implements IPanTool {
fail("cookie未找到"); fail("cookie未找到");
return; return;
} }
Matcher matcher = Pattern.compile(COOKIE_KEY + "([\\w-]+);").matcher(collect.get(0)); Matcher matcher = CSRF_TOKEN_PATTERN.matcher(collect.get(0));
String _t; String _t;
if (matcher.find()) { if (matcher.find()) {
_t = matcher.group(1); _t = matcher.group(1);
@@ -85,7 +86,7 @@ public class PdbTool extends PanBase implements IPanTool {
}) })
.onFailure(handleFail()); .onFailure(handleFail());
} catch (Exception e) { } catch (Exception e) {
e.printStackTrace(); log.error("URL编码异常", e);
} }
}) })
@@ -23,6 +23,16 @@ import java.util.regex.Pattern;
*/ */
public class PodTool extends PanBase { public class PodTool extends PanBase {
private static final int MAX_RESPONSE_BODY_BYTES = 8 * 1024 * 1024;
private static final java.time.Duration REQUEST_TIMEOUT = java.time.Duration.ofSeconds(30);
// 静态共享的 JDK HttpClient 实例避免每次调用创建新实例
private static final HttpClient SHARED_HTTP_CLIENT = HttpClient.newBuilder()
.connectTimeout(java.time.Duration.ofSeconds(10))
.build();
private static volatile WorkerExecutor SHARED_WORKER_EXECUTOR;
private static volatile boolean workerExecutorShutdown = false;
/* /*
* https://1drv.ms/w/s!Alg0feQmCv2rnRFd60DQOmMa-Oh_?e=buaRtp --302-> * https://1drv.ms/w/s!Alg0feQmCv2rnRFd60DQOmMa-Oh_?e=buaRtp --302->
* https://api.onedrive.com/v1.0/drives/abfd0a26e47d3458/items/ABFD0A26E47D3458!3729?authkey=!AF3rQNA6Yxr46H8 * https://api.onedrive.com/v1.0/drives/abfd0a26e47d3458/items/ABFD0A26E47D3458!3729?authkey=!AF3rQNA6Yxr46H8
@@ -45,6 +55,13 @@ public class PodTool extends PanBase {
private static final Pattern redirectUrlRegex = private static final Pattern redirectUrlRegex =
Pattern.compile("resid=(?<cid1>[^!]+)!(?<cid2>[^&]+).+&redeem=(?<redeem>.+).*"); Pattern.compile("resid=(?<cid1>[^!]+)!(?<cid2>[^&]+).+&redeem=(?<redeem>.+).*");
private static final Pattern DOWNLOAD_URL_IN_RESPONSE_PATTERN =
Pattern.compile("\"downloadUrl\":\"(?<url>https?://[^\\s\"]+)");
private static final Pattern ACTION_URL_PATTERN =
Pattern.compile("'action'.+(?<url>https://.+)'\\)");
private static final Pattern TOKEN_PATTERN =
Pattern.compile("inputElem\\.value\\s*=\\s*'([^']+)'");
public PodTool(ShareLinkInfo shareLinkInfo) { public PodTool(ShareLinkInfo shareLinkInfo) {
super(shareLinkInfo); super(shareLinkInfo);
} }
@@ -97,7 +114,7 @@ public class PodTool extends PanBase {
sendHttpRequest(url, token).onSuccess(body -> { sendHttpRequest(url, token).onSuccess(body -> {
Matcher matcher1 = Matcher matcher1 =
Pattern.compile("\"downloadUrl\":\"(?<url>https?://[^\s\"]+)").matcher(body); DOWNLOAD_URL_IN_RESPONSE_PATTERN.matcher(body);
if (matcher1.find()) { if (matcher1.find()) {
// 响应体是 JSON 文本URL 中的 '&' 被转义为 \u0026需要反转义 // 响应体是 JSON 文本URL 中的 '&' 被转义为 \u0026需要反转义
complete(unescapeJsonUnicode(matcher1.group("url"))); complete(unescapeJsonUnicode(matcher1.group("url")));
@@ -121,15 +138,11 @@ public class PodTool extends PanBase {
} }
private String matcherUrl(String html) { private String matcherUrl(String html) {
Matcher urlMatcher = ACTION_URL_PATTERN.matcher(html);
// 正则表达式来匹配 URL
String urlRegex = "'action'.+(?<url>https://.+)'\\)";
Pattern urlPattern = Pattern.compile(urlRegex);
Matcher urlMatcher = urlPattern.matcher(html);
if (urlMatcher.find()) { if (urlMatcher.find()) {
String url = urlMatcher.group("url"); String url = urlMatcher.group("url");
System.out.println("URL: " + url); log.debug("URL: {}", url);
return url; return url;
} }
throw new RuntimeException("URL匹配失败"); throw new RuntimeException("URL匹配失败");
@@ -165,14 +178,11 @@ public class PodTool extends PanBase {
private String matcherToken(String html) { private String matcherToken(String html) {
// 正则表达式来匹配 inputElem.value 中的 Token Matcher tokenMatcher = TOKEN_PATTERN.matcher(html);
String tokenRegex = "inputElem\\.value\\s*=\\s*'([^']+)'";
Pattern tokenPattern = Pattern.compile(tokenRegex);
Matcher tokenMatcher = tokenPattern.matcher(html);
if (tokenMatcher.find()) { if (tokenMatcher.find()) {
String token = tokenMatcher.group(1); String token = tokenMatcher.group(1);
System.out.println("Token: " + token); log.debug("Token: {}***", token.length() > 4 ? token.substring(0, 4) : "***");
return token; return token;
} }
throw new RuntimeException("token匹配失败"); throw new RuntimeException("token匹配失败");
@@ -180,11 +190,8 @@ public class PodTool extends PanBase {
public Future<String> sendHttpRequest2(String token, String redeem) { public Future<String> sendHttpRequest2(String token, String redeem) {
Promise<String> promise = Promise.promise(); Promise<String> promise = Promise.promise();
// 构造 HttpClient
HttpClient client = HttpClient.newHttpClient();
// 构造请求的 URI 和头部信息 // 构造请求的 URI 和头部信息
// https://onedrive.live.com/redir?cid=abfd0a26e47d3458&resid=ABFD0A26E47D3458!4465&ithint=file%2cxlsx&e=Ao2uSU&migratedtospo=true&redeem=aHR0cHM6Ly8xZHJ2Lm1zL3gvYy9hYmZkMGEyNmU0N2QzNDU4L0VWZzBmZVFtQ3YwZ2dLdHhFUUFBQUFBQlRQRWVDMTZfZk1EYk5FTjhEdTRta1E_ZT1BbzJ1U1U
String url = ("https://my.microsoftpersonalcontent.com/_api/v2.0/shares/u!%s/driveItem?$select=content" + String url = ("https://my.microsoftpersonalcontent.com/_api/v2.0/shares/u!%s/driveItem?$select=content" +
".downloadUrl").formatted(redeem); ".downloadUrl").formatted(redeem);
String authorizationHeader = "Badger " + token; String authorizationHeader = "Badger " + token;
@@ -192,15 +199,20 @@ public class PodTool extends PanBase {
// 构建请求 // 构建请求
HttpRequest request = HttpRequest.newBuilder() HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url)) .uri(URI.create(url))
.timeout(REQUEST_TIMEOUT)
.header("Authorization", authorizationHeader) .header("Authorization", authorizationHeader)
.build(); .build();
// 发送请求并处理响应 // 发送请求并处理响应使用共享的 HttpClient
client.sendAsync(request, HttpResponse.BodyHandlers.ofString()) SHARED_HTTP_CLIENT.sendAsync(request, HttpResponse.BodyHandlers.ofByteArray())
.thenApply(response -> { .thenApply(response -> {
System.out.println("Response Status Code: " + response.statusCode()); log.debug("Response Status Code: {}", response.statusCode());
System.out.println("Response Body: " + response.body()); promise.complete(toLimitedString(response.body()));
promise.complete(response.body()); return null;
})
.exceptionally(e -> {
log.error("sendHttpRequest2 请求失败: {}", e.getMessage());
promise.fail(e);
return null; return null;
}); });
@@ -208,18 +220,13 @@ public class PodTool extends PanBase {
} }
public Future<String> sendHttpRequest(String url, String token) { public Future<String> sendHttpRequest(String url, String token) {
// 创建一个 WorkerExecutor 用于异步执行阻塞的 HTTP 请求
WorkerExecutor executor = WebClientVertxInit.get().createSharedWorkerExecutor("http-client-worker");
Promise<String> promise = Promise.promise(); Promise<String> promise = Promise.promise();
executor.executeBlocking(() -> { getWorkerExecutor().executeBlocking(() -> {
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = null;
try { try {
// 构造请求 // 构造请求
request = HttpRequest.newBuilder() HttpRequest request = HttpRequest.newBuilder()
.uri(new URI(url)) .uri(new URI(url))
.timeout(REQUEST_TIMEOUT)
.header("accept", "text/html,application/xhtml+xml,application/xml;q=0.9," + .header("accept", "text/html,application/xhtml+xml,application/xml;q=0.9," +
"image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;" + "image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;" +
"v=b3;q=0.7") "v=b3;q=0.7")
@@ -244,17 +251,49 @@ public class PodTool extends PanBase {
.POST(HttpRequest.BodyPublishers.ofString("badger_token=" + token)) .POST(HttpRequest.BodyPublishers.ofString("badger_token=" + token))
.build(); .build();
// 发起请求并获取响应 // 发起请求并获取响应使用共享的 HttpClient
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString()); HttpResponse<byte[]> response = SHARED_HTTP_CLIENT.send(request, HttpResponse.BodyHandlers.ofByteArray());
// 返回响应体 // 返回响应体
promise.complete(response.body()); promise.complete(toLimitedString(response.body()));
return null; return null;
} catch (URISyntaxException | IOException | InterruptedException e) { } catch (URISyntaxException | IOException | InterruptedException e) {
if (e instanceof InterruptedException) {
Thread.currentThread().interrupt();
}
throw new RuntimeException(e); throw new RuntimeException(e);
} }
}); }).onFailure(promise::fail);
return promise.future(); return promise.future();
} }
private static String toLimitedString(byte[] body) {
if (body.length > MAX_RESPONSE_BODY_BYTES) {
throw new IllegalArgumentException("OneDrive响应体过大: " + body.length + " bytes");
}
return new String(body, java.nio.charset.StandardCharsets.UTF_8);
}
private static WorkerExecutor getWorkerExecutor() {
synchronized (PodTool.class) {
if (workerExecutorShutdown) {
throw new IllegalStateException("OneDrive WorkerExecutor 已关闭");
}
if (SHARED_WORKER_EXECUTOR == null) {
SHARED_WORKER_EXECUTOR = WebClientVertxInit.get().createSharedWorkerExecutor("http-client-worker", 8);
}
return SHARED_WORKER_EXECUTOR;
}
}
public static void shutdownWorkerExecutor() {
synchronized (PodTool.class) {
workerExecutorShutdown = true;
if (SHARED_WORKER_EXECUTOR != null) {
SHARED_WORKER_EXECUTOR.close();
SHARED_WORKER_EXECUTOR = null;
}
}
}
} }
@@ -1,6 +1,5 @@
package cn.qaiu.parser.impl; package cn.qaiu.parser.impl;
import cn.qaiu.WebClientVertxInit;
import cn.qaiu.entity.FileInfo; import cn.qaiu.entity.FileInfo;
import cn.qaiu.entity.ShareLinkInfo; import cn.qaiu.entity.ShareLinkInfo;
import cn.qaiu.parser.PanBase; import cn.qaiu.parser.PanBase;
@@ -11,7 +10,6 @@ import io.vertx.core.MultiMap;
import io.vertx.core.Promise; import io.vertx.core.Promise;
import io.vertx.core.json.JsonObject; import io.vertx.core.json.JsonObject;
import io.vertx.core.json.pointer.JsonPointer; import io.vertx.core.json.pointer.JsonPointer;
import io.vertx.ext.web.client.WebClient;
import io.vertx.uritemplate.UriTemplate; import io.vertx.uritemplate.UriTemplate;
import java.util.List; import java.util.List;
@@ -53,9 +51,8 @@ public class PvyyTool extends PanBase {
@Override @Override
public Future<String> parse() { public Future<String> parse() {
// 请求downcode // 请求downcode - 使用父类的共享 WebClient 而非创建新实例
WebClient.create(WebClientVertxInit.get()) client.getAbs(api + shareLinkInfo.getShareKey())
.getAbs(api + shareLinkInfo.getShareKey())
.send() .send()
.onSuccess(res -> { .onSuccess(res -> {
if (res.statusCode() == 200) { if (res.statusCode() == 200) {
@@ -74,9 +74,9 @@ public class QQTool extends PanBase {
}); });
// 调试匹配的情况 // 调试匹配的情况
System.out.println("文件名称: " + filename); log.debug("文件名称: {}", filename);
System.out.println("文件大小: " + filesize); log.debug("文件大小: {}", filesize);
System.out.println("文件直链: " + fileurl); log.debug("文件直链: {}", fileurl);
// 提交 // 提交
promise.complete(fileurl.replace("\\x26", "&")); promise.complete(fileurl.replace("\\x26", "&"));
@@ -3,9 +3,11 @@ package cn.qaiu.parser.impl;
import cn.qaiu.entity.FileInfo; import cn.qaiu.entity.FileInfo;
import cn.qaiu.entity.ShareLinkInfo; import cn.qaiu.entity.ShareLinkInfo;
import cn.qaiu.parser.PanBase; import cn.qaiu.parser.PanBase;
import cn.qaiu.util.CommonUtils;
import cn.qaiu.util.HeaderUtils; import cn.qaiu.util.HeaderUtils;
import io.vertx.core.Future; import io.vertx.core.Future;
import io.vertx.core.MultiMap; import io.vertx.core.MultiMap;
import io.vertx.core.Promise;
import io.vertx.core.json.JsonArray; import io.vertx.core.json.JsonArray;
import io.vertx.core.json.JsonObject; import io.vertx.core.json.JsonObject;
import org.slf4j.Logger; import org.slf4j.Logger;
@@ -13,27 +15,33 @@ import org.slf4j.LoggerFactory;
import java.net.URLEncoder; import java.net.URLEncoder;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Matcher; import java.util.regex.Matcher;
import java.util.regex.Pattern; import java.util.regex.Pattern;
/** /**
* QQ闪传 <br> * QQ闪传 <br>
* 只能客户端上传 支持Android QQ 9.2.5, MACOS QQ 6.9.78可生成分享链接通过浏览器下载支持超大文件有效期默认7天暂时没找到续期方法<br> * 支持多文件多级目录解析通过 GetFileList API 获取文件列表BatchDownload API 获取下载直链<br>
* 有效期默认7天
*/ */
public class QQscTool extends PanBase { public class QQscTool extends PanBase {
Logger LOG = LoggerFactory.getLogger(QQscTool.class); Logger LOG = LoggerFactory.getLogger(QQscTool.class);
private static final String API_URL = "https://qfile.qq.com/http2rpc/gotrpc/noauth/trpc.qqntv2.richmedia.InnerProxy/BatchDownload"; private static final String BATCH_DOWNLOAD_API =
"https://qfile.qq.com/http2rpc/gotrpc/noauth/trpc.qqntv2.richmedia.InnerProxy/BatchDownload";
private static final MultiMap HEADERS = HeaderUtils.parseHeaders(""" private static final String GET_FILE_LIST_API =
"https://qfile.qq.com/http2rpc/gotrpc/noauth/trpc.file.FileFlashTrans/GetFileList";
private static final MultiMap BATCH_DOWNLOAD_HEADERS = HeaderUtils.parseHeaders("""
Accept-Encoding: gzip, deflate Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9 Accept-Language: zh-CN,zh;q=0.9
Connection: keep-alive Connection: keep-alive
Cookie: uin=9000002; p_uin=9000002 Cookie: uin=9000002; p_uin=9000002
DNT: 1 DNT: 1
Origin: https://qfile.qq.com Origin: https://qfile.qq.com
Referer: https://qfile.qq.com/q/Xolxtv5b4O
Sec-Fetch-Dest: empty Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin Sec-Fetch-Site: same-origin
@@ -46,84 +54,260 @@ public class QQscTool extends PanBase {
x-oidb: {"uint32_command":"0x9248", "uint32_service_type":"4"} x-oidb: {"uint32_command":"0x9248", "uint32_service_type":"4"}
"""); """);
private static final MultiMap GET_FILE_LIST_HEADERS = HeaderUtils.parseHeaders("""
Accept-Encoding: gzip, deflate
Cookie: uin=9000002; p_uin=9000002
Origin: https://qfile.qq.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0
content-type: application/json
x-oidb: {"uint32_command":"0x93d4", "uint32_service_type":"1"}
""");
private static final Pattern FILESET_ID_PATTERN = Pattern.compile(
"fileset_id[^a-f0-9]*([a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12})");
private static final Pattern TITLE_PATTERN = Pattern.compile("<title>(.*?)</title>");
public QQscTool(ShareLinkInfo shareLinkInfo) { public QQscTool(ShareLinkInfo shareLinkInfo) {
super(shareLinkInfo); super(shareLinkInfo);
} }
@Override
public Future<String> parse() { public Future<String> parse() {
String jsonTemplate = """
{"req_head":{"agent":8},"download_info":[{"batch_id":"%s","scene":{"business_type":4,"app_type":22,"scene_type":5},"index_node":{"file_uuid":"%s"},"url_type":2,"download_scene":0}],"scene_type":103}
""";
client.getAbs(shareLinkInfo.getShareUrl()).send(result -> { client.getAbs(shareLinkInfo.getShareUrl()).send(result -> {
if (result.succeeded()) { if (result.failed()) {
String htmlJs = result.result().bodyAsString(); LOG.error("请求失败: {}", result.cause().getMessage());
LOG.debug("获取到的HTML内容: {}", htmlJs); promise.fail(result.cause());
String fileUUID = getFileUUID(htmlJs); return;
String fileName = extractFileNameFromTitle(htmlJs); }
String html = result.result().bodyAsString();
String fileName = extractFileNameFromTitle(html);
if (fileName != null) { if (fileName != null) {
LOG.info("提取到的文件名: {}", fileName);
FileInfo fileInfo = new FileInfo(); FileInfo fileInfo = new FileInfo();
fileInfo.setFileName(fileName); fileInfo.setFileName(fileName);
shareLinkInfo.getOtherParam().put("fileInfo", fileInfo); shareLinkInfo.getOtherParam().put("fileInfo", fileInfo);
} else {
LOG.warn("未能提取到文件名");
} }
if (fileUUID != null) { // 尝试用 GetFileList API 获取第一个文件的下载链接
LOG.info("提取到的文件UUID: {}", fileUUID); String filesetId = extractFilesetId(html);
String formatted = jsonTemplate.formatted(fileUUID, fileUUID); if (filesetId != null) {
JsonObject entries = new JsonObject(formatted); fetchFileList(filesetId, "").onSuccess(fileList -> {
client.postAbs(API_URL) for (int i = 0; i < fileList.size(); i++) {
.putHeaders(HEADERS) JsonObject file = fileList.getJsonObject(i);
.sendJsonObject(entries) if (!file.getBoolean("is_dir", false)) {
.onSuccess(result2 -> { String physicalId = file.getJsonObject("physical").getString("id");
if (result2.statusCode() == 200) { String name = file.getString("name");
JsonObject body = asJson(result2); downloadFile(physicalId, name);
LOG.debug("API响应内容: {}", body.encodePrettily()); return;
// { }
// "retcode": 0, }
// "cost": 132, promise.fail("未找到可下载的文件");
// "message": "", }).onFailure(e -> {
// "error": { LOG.warn("GetFileList 失败,回退到旧解析方式: {}", e.getMessage());
// "message": "", parseLegacy(html, fileName);
// "code": 0 });
// }, } else {
// "data": { parseLegacy(html, fileName);
// "download_rsp": [{ }
});
// download_rsp return promise.future();
if (!body.containsKey("retcode") || body.getInteger("retcode") != 0) { }
promise.fail("API请求失败,错误信息: " + body.encodePrettily());
@Override
public Future<List<FileInfo>> parseFileList() {
Promise<List<FileInfo>> resultPromise = Promise.promise();
String dirId = (String) shareLinkInfo.getOtherParam().get("dirId");
client.getAbs(shareLinkInfo.getShareUrl()).send(result -> {
if (result.failed()) {
resultPromise.fail(result.cause());
return;
}
String html = result.result().bodyAsString();
String filesetId = extractFilesetId(html);
if (filesetId == null) {
resultPromise.fail("无法从页面提取 filesetId");
return;
}
String parentId = dirId != null ? dirId : "";
fetchFileList(filesetId, parentId).onSuccess(fileList -> {
try {
List<FileInfo> list = new ArrayList<>();
String panType = shareLinkInfo.getType();
for (int i = 0; i < fileList.size(); i++) {
JsonObject file = fileList.getJsonObject(i);
FileInfo fileInfo = new FileInfo();
String name = file.getString("name");
String cliFileid = file.getString("cli_fileid");
boolean isDir = file.getBoolean("is_dir", false);
String sizeStr = file.getString("file_size");
fileInfo.setFileName(name)
.setFileId(cliFileid)
.setPanType(panType)
.setSizeStr(sizeStr);
if (isDir) {
fileInfo.setFileType("folder")
.setParserUrl(String.format("%s/v2/getFileList?url=%s&dirId=%s",
getDomainName(),
URLEncoder.encode(shareLinkInfo.getShareUrl(), StandardCharsets.UTF_8),
cliFileid));
} else {
String physicalId = file.getJsonObject("physical").getString("id");
JsonObject paramJson = new JsonObject()
.put("fileId", physicalId)
.put("fileName", name)
.put("cliFileid", cliFileid);
String param = CommonUtils.urlBase64Encode(paramJson.encode());
fileInfo.setFileType("file")
.setParserUrl(String.format("%s/v2/redirectUrl/%s/%s",
getDomainName(), panType, param));
}
list.add(fileInfo);
}
resultPromise.complete(list);
} catch (Exception e) {
resultPromise.fail(e);
}
}).onFailure(resultPromise::fail);
});
return resultPromise.future();
}
@Override
public Future<String> parseById() {
JsonObject paramJson = (JsonObject) shareLinkInfo.getOtherParam().get("paramJson");
String fileId = paramJson.getString("fileId");
String fileName = paramJson.getString("fileName");
Promise<String> p = Promise.promise();
callBatchDownload(fileId, fileName, p);
return p.future();
}
// ========== 内部方法 ==========
/**
* 调用 BatchDownload API 获取单个文件的下载直链
*/
private void downloadFile(String physicalId, String fileName) {
callBatchDownload(physicalId, fileName, promise);
}
private void callBatchDownload(String physicalId, String fileName, Promise<String> p) {
String body = """
{"req_head":{"agent":8},"download_info":[{"batch_id":"%s","scene":{"business_type":4,"app_type":22,"scene_type":5},"index_node":{"file_uuid":"%s"},"url_type":2,"download_scene":0}],"scene_type":103}
""".formatted(physicalId, physicalId);
client.postAbs(BATCH_DOWNLOAD_API)
.putHeaders(BATCH_DOWNLOAD_HEADERS)
.sendJsonObject(new JsonObject(body))
.onSuccess(resp -> {
if (resp.statusCode() != 200) {
p.fail("BatchDownload 请求失败,状态码: " + resp.statusCode());
return;
}
JsonObject respBody = asJson(resp);
if (!respBody.containsKey("retcode") || respBody.getInteger("retcode") != 0) {
p.fail("BatchDownload 请求失败: " + respBody.encodePrettily());
return;
}
JsonArray downloadRsp = respBody.getJsonObject("data").getJsonArray("download_rsp");
if (downloadRsp == null || downloadRsp.isEmpty()) {
p.fail("BatchDownload 响应中缺少 download_rsp");
return; return;
} }
JsonArray downloadRsp = body.getJsonObject("data").getJsonArray("download_rsp");
if (downloadRsp != null && !downloadRsp.isEmpty()) {
String url = downloadRsp.getJsonObject(0).getString("url"); String url = downloadRsp.getJsonObject(0).getString("url");
if (url != null && url.startsWith("&filename=")) {
p.fail("该文件已被和谐");
return;
}
if (fileName != null) { if (fileName != null) {
url = url + "&filename=" + URLEncoder.encode(fileName, StandardCharsets.UTF_8); url = url + "&filename=" + URLEncoder.encode(fileName, StandardCharsets.UTF_8);
} }
promise.complete(url); p.complete(url);
} else { })
promise.fail("API响应中缺少 download_rsp"); .onFailure(e -> {
} LOG.error("BatchDownload 请求异常", e);
} else { p.fail(e);
promise.fail("API请求失败,状态码: " + result2.statusCode());
}
}).onFailure(e -> {
LOG.error("API请求异常", e);
promise.fail(e);
}); });
} else {
LOG.error("未能提取到文件UUID");
promise.fail("未能提取到文件UUID");
} }
} else {
LOG.error("请求失败: {}", result.cause().getMessage());
promise.fail(result.cause());
}
});
return promise.future(); /**
* 调用 GetFileList API 获取指定目录下的文件列表
*/
private Future<JsonArray> fetchFileList(String filesetId, String parentId) {
Promise<JsonArray> p = Promise.promise();
JsonObject body = new JsonObject()
.put("fileset_id", filesetId)
.put("req_infos", new JsonArray()
.add(new JsonObject()
.put("parent_id", parentId)
.put("req_depth", 1)
.put("count", 50)
.put("filter_condition", new JsonObject().put("file_category", 0))
.put("sort_conditions", new JsonArray()
.add(new JsonObject()
.put("sort_field", 0)
.put("sort_order", 0)))))
.put("support_folder_status", true);
// 创建局部副本避免修改静态 MultiMap 导致并发污染
MultiMap headers = MultiMap.caseInsensitiveMultiMap().addAll(GET_FILE_LIST_HEADERS)
.set("Referer", shareLinkInfo.getShareUrl());
client.postAbs(GET_FILE_LIST_API)
.putHeaders(headers)
.sendJsonObject(body)
.onSuccess(resp -> {
if (resp.statusCode() != 200) {
p.fail("GetFileList 请求失败,状态码: " + resp.statusCode());
return;
}
JsonObject respBody = asJson(resp);
if (respBody.getInteger("retcode", -1) != 0) {
p.fail("GetFileList 请求失败: " + respBody.getString("message", "未知错误"));
return;
}
JsonArray fileLists = respBody.getJsonObject("data").getJsonArray("file_lists");
if (fileLists == null || fileLists.isEmpty()) {
p.fail("GetFileList 响应中缺少 file_lists");
return;
}
JsonArray fileList = fileLists.getJsonObject(0).getJsonArray("file_list");
p.complete(fileList != null ? fileList : new JsonArray());
})
.onFailure(e -> {
LOG.error("GetFileList 请求异常", e);
p.fail(e);
});
return p.future();
}
/**
* HTML __NUXT_DATA__ 中提取 fileset_id
*/
String extractFilesetId(String html) {
// Nuxt __NUXT_DATA__ fileset_id 出现在缓存 key 的嵌套 JSON
// 直接匹配 fileset_id 后面最近的 UUID跳过转义引号冒号等非hex字符
Matcher matcher = FILESET_ID_PATTERN.matcher(html);
if (matcher.find()) {
return matcher.group(1);
}
return null;
}
/**
* 旧版解析方式兼容单文件链接通过 HTML 字符串搜索提取 UUID
*/
private void parseLegacy(String html, String fileName) {
String fileUUID = getFileUUID(html);
if (fileUUID == null) {
promise.fail("未能提取到文件UUID");
return;
}
LOG.info("使用旧版解析,提取到的文件UUID: {}", fileUUID);
downloadFile(fileUUID, fileName);
} }
String getFileUUID(String htmlJs) { String getFileUUID(String htmlJs) {
@@ -140,32 +324,22 @@ public class QQscTool extends PanBase {
String extracted = htmlJs.substring(quoteStart, quoteEnd); String extracted = htmlJs.substring(quoteStart, quoteEnd);
LOG.debug("提取结果: {}", extracted); LOG.debug("提取结果: {}", extracted);
return extracted; return extracted;
} else {
LOG.error("未找到结束引号: {}", marker);
} }
} else {
LOG.error("未找到标记: {} 在关键字: {} 之后", marker, keyword);
} }
} else {
LOG.error("未找到关键字: {}", keyword);
} }
return null; return null;
} }
public static String extractFileNameFromTitle(String content) { public static String extractFileNameFromTitle(String content) {
// 匹配<title></title>之间的内容 Matcher matcher = TITLE_PATTERN.matcher(content);
Pattern pattern = Pattern.compile("<title>(.*?)</title>");
Matcher matcher = pattern.matcher(content);
if (matcher.find()) { if (matcher.find()) {
String fullTitle = matcher.group(1); String fullTitle = matcher.group(1);
// "" 分割取前半部分
int sepIndex = fullTitle.indexOf(""); int sepIndex = fullTitle.indexOf("");
if (sepIndex != -1) { if (sepIndex != -1) {
return fullTitle.substring(0, sepIndex); return fullTitle.substring(0, sepIndex);
} }
return fullTitle; // 如果没有分隔符就返回全部 return fullTitle;
} }
return null; return null;
} }
} }
@@ -3,10 +3,11 @@ package cn.qaiu.parser.impl;
import cn.qaiu.entity.FileInfo; import cn.qaiu.entity.FileInfo;
import cn.qaiu.entity.ShareLinkInfo; import cn.qaiu.entity.ShareLinkInfo;
import io.vertx.core.Future; import io.vertx.core.Future;
import java.util.HashMap; import io.vertx.core.buffer.Buffer;
import java.util.Map; import io.vertx.core.json.JsonObject;
import java.util.regex.Matcher;
import java.util.regex.Pattern; import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
public class QQwTool extends QQTool { public class QQwTool extends QQTool {
@@ -16,54 +17,51 @@ public class QQwTool extends QQTool {
@Override @Override
public Future<String> parse() { public Future<String> parse() {
client.getAbs(shareLinkInfo.getShareUrl()).send().onSuccess(res -> { String k = shareLinkInfo.getShareKey();
String html = res.bodyAsString(); String postBody = "f=json&k=" + URLEncoder.encode(k, StandardCharsets.UTF_8);
Map<String, String> stringStringMap = extractVariables(html);
String url = stringStringMap.get("url");
String fn = stringStringMap.get("filename");
String size = stringStringMap.get("filesize");
String createBy = stringStringMap.get("nick");
FileInfo fileInfo = new FileInfo().setFileName(fn).setSize(Long.parseLong(size)).setCreateBy(createBy);
shareLinkInfo.getOtherParam().put("fileInfo", fileInfo);
if (url != null) {
String url302 = url.replace("\\x26", "&");
promise.complete(url302);
/* client.postAbs("https://wx.mail.qq.com/s")
clientNoRedirects.getAbs(url302).send().onSuccess(res2 -> { .putHeader("Content-Type", "application/x-www-form-urlencoded")
MultiMap headers = res2.headers(); .putHeader("Accept", "application/json, text/plain, */*")
if (headers.contains("Location")) { .putHeader("Referer", shareLinkInfo.getShareUrl())
promise.complete(headers.get("Location")); .putHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
} else { + "(KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 Edg/134.0.0.0")
fail("找不到重定向URL"); .sendBuffer(Buffer.buffer(postBody))
.onSuccess(res -> {
try {
JsonObject data = asJson(res);
JsonObject head = data.getJsonObject("head");
if (head == null || head.getInteger("ret", -1) != 0) {
String msg = head != null ? head.getString("msg", "未知错误") : "未知错误";
fail("API错误: " + msg);
return;
} }
}).onFailure(handleFail()); JsonObject body = data.getJsonObject("body");
*/ if (body == null) {
} else { fail("文件信息为空");
return;
}
String url = body.getString("url");
String fn = body.getString("name", "");
long size = body.getLong("size", 0L);
if (url == null || url.isEmpty()) {
fail("分享链接解析失败, 可能是链接失效"); fail("分享链接解析失败, 可能是链接失效");
return;
}
FileInfo fileInfo = new FileInfo().setFileName(fn).setSize(size);
shareLinkInfo.getOtherParam().put("fileInfo", fileInfo);
String url302 = url.replace("\\x26", "&");
complete(url302);
} catch (Exception e) {
fail(e, "解析响应失败");
} }
}).onFailure(handleFail()); }).onFailure(handleFail());
return promise.future(); return promise.future();
} }
private Map<String, String> extractVariables(String jsCode) {
Map<String, String> variables = new HashMap<>();
// 正则表达式匹配 var 变量定义
String regex = "\\s+var\\s+(\\w+)\\s*=\\s*(?:\"([^\"]*)\"|'([^']*)'|([^;\\r\\n]*))";
Pattern p = Pattern.compile(regex);
Matcher m = p.matcher(jsCode);
while (m.find()) {
String name = m.group(1);
String value = m.group(2) != null ? m.group(2)
: m.group(3) != null ? m.group(3)
: m.group(4);
variables.put(name, value);
}
return variables;
}
} }
File diff suppressed because it is too large Load Diff
@@ -265,13 +265,8 @@ public class UcTool extends PanBase {
return; return;
} }
String downloadUrl = dataList.getJsonObject(0).getString("download_url"); String downloadUrl = dataList.getJsonObject(0).getString("download_url");
// UC网盘需要配合aria2下载保存下载请求头 // UC 需配合下载器 cookie保存下载请求头
Map<String, String> downloadHeaders = new HashMap<>(); completeWithMeta(downloadUrl, buildDownloadHeaders(null));
// 将header转换为Map 只需要包含cookie,user-agent,referer
downloadHeaders.put(HttpHeaders.COOKIE.toString(), header.get(HttpHeaders.COOKIE));
downloadHeaders.put(HttpHeaders.USER_AGENT.toString(), header.get(HttpHeaders.USER_AGENT));
downloadHeaders.put(HttpHeaders.REFERER.toString(), "https://fast.uc.cn/");
completeWithMeta(downloadUrl, downloadHeaders);
} catch (Exception e) { } catch (Exception e) {
fail("解析 UC 下载链接失败: " + e.getMessage()); fail("解析 UC 下载链接失败: " + e.getMessage());
} }
@@ -466,29 +461,40 @@ public class UcTool extends PanBase {
if (shareFidToken != null) { if (shareFidToken != null) {
extParams.put("share_fid_token", shareFidToken); extParams.put("share_fid_token", shareFidToken);
} }
extParams.put("needDownloader", true);
Map<String, String> dlHeaders = new HashMap<>();
String listCookie = header.get(HttpHeaders.COOKIE);
if (listCookie != null && !listCookie.isEmpty()) {
dlHeaders.put(HttpHeaders.COOKIE.toString(), listCookie);
}
dlHeaders.put(HttpHeaders.USER_AGENT.toString(), header.get(HttpHeaders.USER_AGENT));
dlHeaders.put(HttpHeaders.REFERER.toString(), "https://fast.uc.cn/");
extParams.put("downloadHeaders", dlHeaders);
fileInfo.setExtParameters(extParams); fileInfo.setExtParameters(extParams);
// 设置解析URL用于下载 // 设置解析URL用于下载
JsonObject paramJson = new JsonObject(extParams); JsonObject paramJson = new JsonObject(extParams);
paramJson.put("fileName", fileName);
String param = CommonUtils.urlBase64Encode(paramJson.encode()); String param = CommonUtils.urlBase64Encode(paramJson.encode());
fileInfo.setParserUrl(String.format("%s/v2/redirectUrl/%s/%s", // 透传 auth避免下载/转存时变成 guest
getDomainName(), shareLinkInfo.getType(), param)); fileInfo.setParserUrl(appendAuthQuery(String.format("%s/v2/redirectUrl/%s/%s",
getDomainName(), shareLinkInfo.getType(), param)));
} else { } else {
// 文件夹 // 文件夹
fileInfo.setFileType("folder"); fileInfo.setFileType("folder");
fileInfo.setSize(0L); fileInfo.setSize(0L);
fileInfo.setSizeStr("0B"); fileInfo.setSizeStr("0B");
// 设置目录解析URL用于递归解析子目录 // 递归子目录须透传 auth否则会丢失认证
// URL 参数进行编码确保特殊字符正确传递
try { try {
String encodedUrl = URLEncoder.encode(shareLinkInfo.getShareUrl(), StandardCharsets.UTF_8.toString()); String encodedUrl = URLEncoder.encode(shareLinkInfo.getShareUrl(), StandardCharsets.UTF_8.toString());
String encodedDirId = URLEncoder.encode(fid, StandardCharsets.UTF_8.toString()); String encodedDirId = URLEncoder.encode(fid, StandardCharsets.UTF_8.toString());
String encodedStoken = URLEncoder.encode(stoken, StandardCharsets.UTF_8.toString()); String encodedStoken = URLEncoder.encode(stoken, StandardCharsets.UTF_8.toString());
fileInfo.setParserUrl(String.format("%s/v2/getFileList?url=%s&dirId=%s&stoken=%s", fileInfo.setParserUrl(appendAuthQuery(String.format(
getDomainName(), encodedUrl, encodedDirId, encodedStoken)); "%s/v2/getFileList?url=%s&dirId=%s&stoken=%s",
getDomainName(), encodedUrl, encodedDirId, encodedStoken)));
} catch (Exception e) { } catch (Exception e) {
// 如果编码失败使用原始值 fileInfo.setParserUrl(appendAuthQuery(String.format(
fileInfo.setParserUrl(String.format("%s/v2/getFileList?url=%s&dirId=%s&stoken=%s", "%s/v2/getFileList?url=%s&dirId=%s&stoken=%s",
getDomainName(), shareLinkInfo.getShareUrl(), fid, stoken)); getDomainName(), shareLinkInfo.getShareUrl(), fid, stoken)));
} }
} }
@@ -511,6 +517,9 @@ public class UcTool extends PanBase {
return promise.future(); return promise.future();
} }
// 会话无 cookie 回退使用入口参数中已带的 cookie
ensureCookieFromParam(paramJson);
String fid = paramJson.getString("fid"); String fid = paramJson.getString("fid");
String pwdId = paramJson.getString("pwd_id"); String pwdId = paramJson.getString("pwd_id");
String stoken = paramJson.getString("stoken"); String stoken = paramJson.getString("stoken");
@@ -554,6 +563,11 @@ public class UcTool extends PanBase {
promise.fail("未找到下载链接"); promise.fail("未找到下载链接");
return; return;
} }
// 存储下载请求头供目录解析 getFileDownInfo 接口使用
// 优先用当前会话 cookie缺失时回退入口参数中已带的 cookie
Map<String, String> downloadHeaders = buildDownloadHeaders(paramJson);
shareLinkInfo.getOtherParam().put("downloadHeaders", downloadHeaders);
shareLinkInfo.getOtherParam().put("fileName", paramJson.getString("fileName", ""));
promise.complete(downloadUrl); promise.complete(downloadUrl);
} catch (Exception e) { } catch (Exception e) {
promise.fail("解析 UC 下载链接失败: " + e.getMessage()); promise.fail("解析 UC 下载链接失败: " + e.getMessage());
@@ -564,6 +578,53 @@ public class UcTool extends PanBase {
return promise.future(); return promise.future();
} }
/**
* 会话无 cookie 从入口参数 downloadHeaders 回填到请求头
*/
private void ensureCookieFromParam(JsonObject paramJson) {
String cookie = header.get(HttpHeaders.COOKIE);
if (cookie != null && !cookie.isEmpty()) {
return;
}
String paramCookie = extractCookieFromParam(paramJson);
if (paramCookie != null && !paramCookie.isEmpty()) {
header.set(HttpHeaders.COOKIE, CookieUtils.filterUcQuarkCookie(paramCookie));
}
}
private static String extractCookieFromParam(JsonObject paramJson) {
if (paramJson == null) {
return null;
}
JsonObject paramHeaders = paramJson.getJsonObject("downloadHeaders");
if (paramHeaders == null) {
return null;
}
String cookie = paramHeaders.getString("cookie");
return cookie != null ? cookie : paramHeaders.getString("Cookie");
}
/**
* 构建下载请求头会话 cookie 优先缺失时回退入口参数中的 downloadHeaders
*/
private Map<String, String> buildDownloadHeaders(JsonObject paramJson) {
Map<String, String> downloadHeaders = new HashMap<>();
String cookie = header.get(HttpHeaders.COOKIE);
if (cookie == null || cookie.isEmpty()) {
cookie = extractCookieFromParam(paramJson);
}
if (cookie != null && !cookie.isEmpty()) {
downloadHeaders.put(HttpHeaders.COOKIE.toString(), cookie);
}
String ua = header.get(HttpHeaders.USER_AGENT);
if (ua == null || ua.isEmpty()) {
ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36";
}
downloadHeaders.put(HttpHeaders.USER_AGENT.toString(), ua);
downloadHeaders.put(HttpHeaders.REFERER.toString(), "https://fast.uc.cn/");
return downloadHeaders;
}
// public static void main(String[] args) { // public static void main(String[] args) {
// // https://drive.uc.cn/s/12450d1694844?public=1 // // https://drive.uc.cn/s/12450d1694844?public=1
// new UcTool(ShareLinkInfo.newBuilder().shareKey("12450d1694844").build()).parse().onSuccess( // new UcTool(ShareLinkInfo.newBuilder().shareKey("12450d1694844").build()).parse().onSuccess(
@@ -67,11 +67,7 @@ public class WsTool extends PanBase {
String filepid = asJson(res2).getJsonObject("data").getString("ufileid"); // 文件夹pid String filepid = asJson(res2).getJsonObject("data").getString("ufileid"); // 文件夹pid
String filebid = asJson(res2).getJsonObject("data").getString("boxid"); // 文件夹bid String filebid = asJson(res2).getJsonObject("data").getString("boxid"); // 文件夹bid
// 调试输出文件夹信息 log.debug("文件夹期限: {}, 大小: {}, pid: {}, bid: {}", filetime, filesize, filepid, filebid);
System.out.println("文件夹期限: " + filetime);
System.out.println("文件夹大小: " + filesize);
System.out.println("文件夹pid: " + filepid);
System.out.println("文件夹bid: " + filebid);
// 获取文件信息 // 获取文件信息
httpClient.postAbs(SHARE_URL_API + "ufile/list").putHeaders(headers) httpClient.postAbs(SHARE_URL_API + "ufile/list").putHeaders(headers)
@@ -97,9 +93,7 @@ public class WsTool extends PanBase {
String filefid = asJson(res3).getJsonObject("data") String filefid = asJson(res3).getJsonObject("data")
.getJsonArray("fileList").getJsonObject(0).getString("fid"); // 文件fid .getJsonArray("fileList").getJsonObject(0).getString("fid"); // 文件fid
// 调试输出文件信息 log.debug("文件名称: {}, fid: {}", filename, filefid);
System.out.println("文件名称: " + filename);
System.out.println("文件fid: " + filefid);
// 检查文件是否失效 // 检查文件是否失效
httpClient.postAbs(SHARE_URL_API + "dl/sign").putHeaders(headers) httpClient.postAbs(SHARE_URL_API + "dl/sign").putHeaders(headers)
@@ -114,8 +108,7 @@ public class WsTool extends PanBase {
// 获取直链 // 获取直链
String fileurl = asJson(res4).getJsonObject("data").getString("url"); String fileurl = asJson(res4).getJsonObject("data").getString("url");
// 调试输出文件直链 log.debug("文件直链: {}", fileurl);
System.out.println("文件直链: " + fileurl);
if (!fileurl.equals("")) { if (!fileurl.equals("")) {
promise.complete(URLDecoder.decode(fileurl, StandardCharsets.UTF_8)); promise.complete(URLDecoder.decode(fileurl, StandardCharsets.UTF_8));
@@ -3,8 +3,10 @@ package cn.qaiu.parser.impl;
import cn.qaiu.entity.FileInfo; import cn.qaiu.entity.FileInfo;
import cn.qaiu.entity.ShareLinkInfo; import cn.qaiu.entity.ShareLinkInfo;
import cn.qaiu.parser.PanBase; import cn.qaiu.parser.PanBase;
import cn.qaiu.parser.TokenCache;
import cn.qaiu.util.CommonUtils; import cn.qaiu.util.CommonUtils;
import cn.qaiu.util.FileSizeConverter; import cn.qaiu.util.FileSizeConverter;
import cn.qaiu.util.YeShareHostUtil;
import io.vertx.core.Future; import io.vertx.core.Future;
import io.vertx.core.MultiMap; import io.vertx.core.MultiMap;
import io.vertx.core.Promise; import io.vertx.core.Promise;
@@ -19,7 +21,11 @@ import org.apache.commons.lang3.StringUtils;
import java.net.MalformedURLException; import java.net.MalformedURLException;
import java.time.OffsetDateTime; import java.time.OffsetDateTime;
import java.time.format.DateTimeFormatter; import java.time.format.DateTimeFormatter;
import java.util.*; import java.util.ArrayList;
import java.util.Base64;
import java.util.List;
import java.util.Map;
import java.util.Random;
import java.util.zip.CRC32; import java.util.zip.CRC32;
import static cn.qaiu.util.RandomStringGenerator.gen36String; import static cn.qaiu.util.RandomStringGenerator.gen36String;
@@ -32,24 +38,22 @@ import static cn.qaiu.util.RandomStringGenerator.gen36String;
*/ */
public class Ye2Tool extends PanBase { public class Ye2Tool extends PanBase {
public static final String SHARE_URL_PREFIX = "https://www.123pan.com/s/"; private static final String API_BASE = "https://api.123278.com";
public static final String FIRST_REQUEST_URL = SHARE_URL_PREFIX + "{key}.html"; private static final String GET_SHARE_INFO_URL = API_BASE + "/b/api/share/get?limit=100&next=1&orderBy=share_id&orderDirection=desc&shareKey={shareKey}&SharePwd={pwd}&ParentFileId={ParentFileId}&Page=1";
private static final String GET_SHARE_INFO_URL = "https://www.123pan.com/b/api/share/get?limit=100&next=1&orderBy=share_id&orderDirection=desc&shareKey={shareKey}&SharePwd={pwd}&ParentFileId={ParentFileId}&Page=1"; private static final String DOWNLOAD_API_URL = API_BASE + "/b/api/file/download_info";
private static final String DOWNLOAD_API_URL = "https://www.123pan.com/b/api/file/download_info"; private static final String DOWNLOAD_API_V2_BASE = "https://api.123278.com";
private static final String BATCH_DOWNLOAD_API_URL = "https://www.123pan.com/b/api/file/batch_download_share_info"; private static final String DOWNLOAD_API_V2_PATH = "/b/api/v2/share/download/info";
private static final String BATCH_DOWNLOAD_API_URL = API_BASE + "/b/api/file/batch_download_share_info";
private static final String LOGIN_URL = "https://login.123pan.com/api/user/sign_in"; private static final String LOGIN_URL = "https://login.123pan.com/api/user/sign_in";
// 字符映射表
private static final String CHAR_MAP = "adefghlmyijnopkqrstubcvwsz"; private static final String CHAR_MAP = "adefghlmyijnopkqrstubcvwsz";
private final MultiMap header = MultiMap.caseInsensitiveMultiMap(); private final MultiMap header = MultiMap.caseInsensitiveMultiMap();
private final String cacheKey;
// Token管理
private static String ssoToken;
private static long tokenExpireTime = 0L; // 毫秒时间戳
public Ye2Tool(ShareLinkInfo shareLinkInfo) { public Ye2Tool(ShareLinkInfo shareLinkInfo) {
super(shareLinkInfo); super(shareLinkInfo);
this.cacheKey = TokenCache.key("ye2", resolveAccountId());
header.set("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6"); header.set("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6");
header.set("App-Version", "55"); header.set("App-Version", "55");
header.set("Cache-Control", "no-cache"); header.set("Cache-Control", "no-cache");
@@ -65,16 +69,62 @@ public class Ye2Tool extends PanBase {
header.set("Content-Type", "application/json"); header.set("Content-Type", "application/json");
} }
/** private String resolveAccountId() {
* 判断 token 是否过期 String accountId = "_default";
*/ if (!shareLinkInfo.getOtherParam().containsKey("auths")) {
private boolean isTokenExpired() { return accountId;
return System.currentTimeMillis() > tokenExpireTime - 60_000; // 提前1分钟刷新 }
MultiMap auths = (MultiMap) shareLinkInfo.getOtherParam().get("auths");
if (auths.contains("_configId")) {
accountId = auths.get("_configId");
} else if (auths.contains("username")) {
accountId = auths.get("username");
} else {
String token = resolveProvidedToken(auths);
if (StringUtils.isNotEmpty(token)) {
accountId = token.substring(0, Math.min(16, token.length()));
}
}
return accountId;
} }
/** /**
* 计算CRC32并转换为16进制字符串 * 从配置/临时认证参数中取出可直接使用的 Bearer token
* 兼容两种 key
* - tokenURLParamUtil 处理 auth= 临时参数(authType=accesstoken/authorization)时写入的字段名
* - authorizationapp-dev.yml 静态配置中更符合直觉的写法auths.ye.authorization: xxx
*/ */
private String resolveProvidedToken(MultiMap auths) {
if (auths == null) {
return null;
}
String token = auths.get("token");
if (StringUtils.isEmpty(token)) {
token = auths.get("authorization");
}
return stripBearerPrefix(token);
}
/**
* 用户/捐赠账号提供的 token 经常是从浏览器开发者工具里连同 "Bearer " 前缀一起复制过来的
* 如果不做清理后续 header.set("Authorization", "Bearer " + token) 会拼出
* "Bearer Bearer xxx"导致123网盘接口报错 tokenstring should not contain 'bearer '
*/
private String stripBearerPrefix(String token) {
if (StringUtils.isBlank(token)) {
return token;
}
String trimmed = token.trim();
if (trimmed.regionMatches(true, 0, "Bearer ", 0, 7)) {
trimmed = trimmed.substring(7).trim();
}
return trimmed;
}
private boolean isTokenExpired() {
return TokenCache.isExpired(cacheKey);
}
private String crc32(String data) { private String crc32(String data) {
CRC32 crc32 = new CRC32(); CRC32 crc32 = new CRC32();
crc32.update(data.getBytes()); crc32.update(data.getBytes());
@@ -82,105 +132,58 @@ public class Ye2Tool extends PanBase {
return String.format("%08x", value); return String.format("%08x", value);
} }
/**
* 16进制转10进制
*/
private long hexToInt(String hexStr) { private long hexToInt(String hexStr) {
return Long.parseLong(hexStr, 16); return Long.parseLong(hexStr, 16);
} }
/**
* 123盘的URL加密算法
* 参考Python代码中的encode123函数
*
* @param url 请求路径
* @param way 平台标识"android"
* @param version 版本号"55"
* @param timestamp 时间戳毫秒
* @return 加密后的URL参数格式?{y}={time_long}-{a}-{final_crc}
*/
private String encode123(String url, String way, String version, String timestamp) { private String encode123(String url, String way, String version, String timestamp) {
Random random = new Random(); Random random = new Random();
// 生成随机数 a = int(10000000 * random.randint(1, 10000000) / 10000)
int randomInt = random.nextInt(10000000) + 1; int randomInt = random.nextInt(10000000) + 1;
long a = (10000000L * randomInt) / 10000; long a = (10000000L * randomInt) / 10000;
// 将时间戳转换为时间格式
long timeLong = Long.parseLong(timestamp) / 1000; long timeLong = Long.parseLong(timestamp) / 1000;
java.time.LocalDateTime dateTime = java.time.Instant.ofEpochSecond(timeLong) java.time.LocalDateTime dateTime = java.time.Instant.ofEpochSecond(timeLong)
.atZone(java.time.ZoneId.systemDefault()) .atZone(java.time.ZoneId.systemDefault())
.toLocalDateTime(); .toLocalDateTime();
String timeStr = dateTime.format(DateTimeFormatter.ofPattern("yyyyMMddHHmm")); String timeStr = dateTime.format(DateTimeFormatter.ofPattern("yyyyMMddHHmm"));
// 根据时间字符串生成g
StringBuilder g = new StringBuilder(); StringBuilder g = new StringBuilder();
for (char c : timeStr.toCharArray()) { for (char c : timeStr.toCharArray()) {
int digit = Character.getNumericValue(c); int digit = Character.getNumericValue(c);
if (digit == 0) { if (digit == 0) {
g.append(CHAR_MAP.charAt(0)); g.append(CHAR_MAP.charAt(0));
} else { } else {
// 数字1对应索引0数字2对应索引1以此类推
g.append(CHAR_MAP.charAt(digit - 1)); g.append(CHAR_MAP.charAt(digit - 1));
} }
} }
// 计算y值CRC32的十进制
String y = String.valueOf(hexToInt(crc32(g.toString()))); String y = String.valueOf(hexToInt(crc32(g.toString())));
// 计算最终的CRC32
String finalCrcInput = String.format("%d|%d|%s|%s|%s|%s", timeLong, a, url, way, version, y); String finalCrcInput = String.format("%d|%d|%s|%s|%s|%s", timeLong, a, url, way, version, y);
String finalCrc = String.valueOf(hexToInt(crc32(finalCrcInput))); String finalCrc = String.valueOf(hexToInt(crc32(finalCrcInput)));
// 返回加密后的URL参数
return String.format("?%s=%d-%d-%s", y, timeLong, a, finalCrc); return String.format("?%s=%d-%d-%s", y, timeLong, a, finalCrc);
} }
public Future<String> parse() { public Future<String> parse() {
Future<String> tokenFuture; Future<String> tokenFuture = resolveTokenFuture();
// 检查是否直接提供了token
MultiMap auths = (MultiMap) shareLinkInfo.getOtherParam().get("auths");
if (auths != null && auths.contains("token")) {
String providedToken = auths.get("token");
if (StringUtils.isNotEmpty(providedToken)) {
ssoToken = providedToken;
tokenFuture = Future.succeededFuture(providedToken);
} else {
// 如果没有提供token尝试登录
if (ssoToken == null || isTokenExpired()) {
tokenFuture = loginAndGetToken();
} else {
tokenFuture = Future.succeededFuture(ssoToken);
}
}
} else {
// 如果没有提供token尝试登录
if (ssoToken == null || isTokenExpired()) {
tokenFuture = loginAndGetToken();
} else {
tokenFuture = Future.succeededFuture(ssoToken);
}
}
// 1. 登录获取 sso-token 或使用提供的token
tokenFuture.onSuccess(token -> { tokenFuture.onSuccess(token -> {
if (!token.equals("nologin")) { if (!token.equals("nologin")) {
// 2. 设置 header TokenCache.putToken(cacheKey, token);
ssoToken = token;
header.set("Authorization", "Bearer " + token); header.set("Authorization", "Bearer " + token);
} }
final String dataKey = shareLinkInfo.getShareKey().replace(".html", ""); final String dataKey = YeShareHostUtil.normalizeShareKey(shareLinkInfo.getShareKey());
final String pwd = shareLinkInfo.getSharePassword(); final String pwd = shareLinkInfo.getSharePassword();
final String shareOrigin = resolveYeShareOrigin(dataKey);
final String shareReferer = buildShareReferer(shareOrigin, dataKey, pwd);
// 3. 获取分享信息
client.getAbs(UriTemplate.of(GET_SHARE_INFO_URL)) client.getAbs(UriTemplate.of(GET_SHARE_INFO_URL))
.setTemplateParam("shareKey", dataKey) .setTemplateParam("shareKey", dataKey)
.setTemplateParam("pwd", StringUtils.isEmpty(pwd) ? "" : pwd) .setTemplateParam("pwd", StringUtils.isEmpty(pwd) ? "" : pwd)
.setTemplateParam("ParentFileId", "0") .setTemplateParam("ParentFileId", "0")
.putHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36") .putHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36")
.putHeader("Referer", "https://www.123pan.com/") .putHeader("Referer", shareReferer)
.putHeader("Origin", "https://www.123pan.com") .putHeader("Origin", shareOrigin)
.send() .send()
.onSuccess(res -> { .onSuccess(res -> {
JsonObject shareInfoJson = asJson(res); JsonObject shareInfoJson = asJson(res);
@@ -200,35 +203,47 @@ public class Ye2Tool extends PanBase {
return; return;
} }
// 获取第一个文件信息
JsonObject fileInfo = data.getJsonArray("InfoList").getJsonObject(0); JsonObject fileInfo = data.getJsonArray("InfoList").getJsonObject(0);
// 检查是否需要登录
if (token.equals("nologin")) { if (token.equals("nologin")) {
fail("该分享需要登录才能下载,请提供账号密码或token"); fail("该分享需要登录才能下载,请配置认证信息");
return; return;
} }
// 判断是否为文件夹: Type: 1为文件夹, 0为文件
if (fileInfo.getInteger("Type", 0) == 1) { if (fileInfo.getInteger("Type", 0) == 1) {
// 4. 获取文件夹打包下载链接
getZipDownUrl(client, fileInfo); getZipDownUrl(client, fileInfo);
} else { } else {
// 4. 获取文件下载链接
getDownUrl(client, fileInfo); getDownUrl(client, fileInfo);
} }
}) })
.onFailure(this.handleFail(GET_SHARE_INFO_URL)); .onFailure(this.handleFail(GET_SHARE_INFO_URL));
}).onFailure(err -> { }).onFailure(err -> fail("123盘解析异常: {}", err.getMessage()));
fail("登录获取token失败: {}", err.getMessage());
});
return promise.future(); return promise.future();
} }
/** private Future<String> resolveTokenFuture() {
* 登录并获取token MultiMap auths = (MultiMap) shareLinkInfo.getOtherParam().get("auths");
*/ if (auths != null) {
// 当同时提供了用户名+密码时优先走真实登录流程
// 前端/auth参数中 authType=password|username_password URLParamUtil 出于兼容旧解析器的目的
// 会把 authToken(用户名) 同时写入 token 字段如果这里无条件信任 token会把用户名当成
// Bearer token 使用导致真实的账号密码登录被跳过
boolean hasCredential = StringUtils.isNotEmpty(auths.get("username"))
&& StringUtils.isNotEmpty(auths.get("password"));
String providedToken = resolveProvidedToken(auths);
if (!hasCredential && StringUtils.isNotEmpty(providedToken)) {
TokenCache.putToken(cacheKey, providedToken);
return Future.succeededFuture(providedToken);
}
}
String cached = TokenCache.getToken(cacheKey);
if (cached == null || isTokenExpired()) {
return loginAndGetToken();
}
return Future.succeededFuture(cached);
}
private Future<String> loginAndGetToken() { private Future<String> loginAndGetToken() {
MultiMap auths = (MultiMap) shareLinkInfo.getOtherParam().get("auths"); MultiMap auths = (MultiMap) shareLinkInfo.getOtherParam().get("auths");
if (auths == null) { if (auths == null) {
@@ -275,16 +290,17 @@ public class Ye2Tool extends PanBase {
promise.fail("未获取到token"); promise.fail("未获取到token");
return; return;
} }
ssoToken = data.getString("token"); String ssoToken = data.getString("token");
String expireStr = data.getString("expire"); String expireStr = data.getString("expire");
// 解析过期时间 long expireMs;
if (StringUtils.isNotEmpty(expireStr)) { if (StringUtils.isNotEmpty(expireStr)) {
tokenExpireTime = OffsetDateTime.parse(expireStr) expireMs = OffsetDateTime.parse(expireStr)
.toInstant().toEpochMilli(); .toInstant().toEpochMilli() - 60_000;
} else { } else {
// 如果没有过期时间默认1小时后过期 expireMs = System.currentTimeMillis() + 3600_000;
tokenExpireTime = System.currentTimeMillis() + 3600_000;
} }
TokenCache.putToken(cacheKey, ssoToken);
TokenCache.putExpire(cacheKey, expireMs);
log.info("登录成功,token: {}", ssoToken); log.info("登录成功,token: {}", ssoToken);
promise.complete(ssoToken); promise.complete(ssoToken);
}) })
@@ -292,13 +308,34 @@ public class Ye2Tool extends PanBase {
return promise.future(); return promise.future();
} }
/**
* 获取下载链接使用Android平台API
*/
private void getDownUrl(WebClient client, JsonObject fileInfo) { private void getDownUrl(WebClient client, JsonObject fileInfo) {
setFileInfo(fileInfo); setFileInfo(fileInfo);
// 构建请求数据 String normalizedShareKey = YeShareHostUtil.normalizeShareKey(shareLinkInfo.getShareKey());
if (StringUtils.isNotEmpty(normalizedShareKey)) {
JsonObject v2Body = new JsonObject()
.put("ShareKey", normalizedShareKey)
.put("FileID", fileInfo.getInteger("FileId"))
.put("S3keyFlag", fileInfo.getString("S3KeyFlag"))
.put("Size", fileInfo.getLong("Size"))
.put("Etag", fileInfo.getString("Etag"));
requestShareV2Download(normalizedShareKey, v2Body).onSuccess(v2Url -> {
if (StringUtils.isNotEmpty(v2Url)) {
complete(v2Url);
return;
}
requestLegacyDownUrl(client, fileInfo);
}).onFailure(err -> {
log.warn("Ye2 v2分享下载接口失败,回退旧接口: {}", err.getMessage());
requestLegacyDownUrl(client, fileInfo);
});
return;
}
requestLegacyDownUrl(client, fileInfo);
}
private void requestLegacyDownUrl(WebClient client, JsonObject fileInfo) {
JsonObject jsonObject = new JsonObject(); JsonObject jsonObject = new JsonObject();
jsonObject.put("driveId", 0); jsonObject.put("driveId", 0);
jsonObject.put("etag", fileInfo.getString("Etag")); jsonObject.put("etag", fileInfo.getString("Etag"));
@@ -308,7 +345,6 @@ public class Ye2Tool extends PanBase {
jsonObject.put("size", fileInfo.getLong("Size")); jsonObject.put("size", fileInfo.getLong("Size"));
jsonObject.put("type", 0); jsonObject.put("type", 0);
// 使用encode123加密URL参数
String timestamp = String.valueOf(System.currentTimeMillis()); String timestamp = String.valueOf(System.currentTimeMillis());
String encryptedParams = encode123("/b/api/file/download_info", "android", "55", timestamp); String encryptedParams = encode123("/b/api/file/download_info", "android", "55", timestamp);
String apiUrl = DOWNLOAD_API_URL + encryptedParams; String apiUrl = DOWNLOAD_API_URL + encryptedParams;
@@ -318,21 +354,98 @@ public class Ye2Tool extends PanBase {
HttpRequest<Buffer> bufferHttpRequest = client.postAbs(apiUrl); HttpRequest<Buffer> bufferHttpRequest = client.postAbs(apiUrl);
bufferHttpRequest.putHeader("platform", "android"); bufferHttpRequest.putHeader("platform", "android");
bufferHttpRequest.putHeader("App-Version", "55"); bufferHttpRequest.putHeader("App-Version", "55");
bufferHttpRequest.putHeader("Authorization", "Bearer " + ssoToken); bufferHttpRequest.putHeader("Authorization", "Bearer " + TokenCache.getToken(cacheKey));
bufferHttpRequest.putHeader("User-Agent", "Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36"); bufferHttpRequest.putHeader("User-Agent", "Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36");
bufferHttpRequest.putHeader("Content-Type", "application/json"); bufferHttpRequest.putHeader("Content-Type", "application/json");
bufferHttpRequest bufferHttpRequest
.sendJsonObject(jsonObject) .sendJsonObject(jsonObject)
.onSuccess(res2 -> { .onSuccess(res2 -> handleDownloadUrlResponse(client, asJson(res2), "Ye2"))
JsonObject downURLJson = asJson(res2); .onFailure(err -> fail("下载接口失败: " + err.getMessage()));
}
private Future<String> requestShareV2Download(String shareKey, JsonObject body) {
Promise<String> promise = Promise.promise();
String timestamp = String.valueOf(System.currentTimeMillis());
String encryptedParams = encode123(DOWNLOAD_API_V2_PATH, "web", "3", timestamp);
String apiUrl = DOWNLOAD_API_V2_BASE + DOWNLOAD_API_V2_PATH + encryptedParams;
String shareOrigin = resolveYeShareOrigin(shareKey);
String shareReferer = buildShareReferer(shareOrigin, shareKey, shareLinkInfo.getSharePassword());
HttpRequest<Buffer> request = client.postAbs(apiUrl);
request.putHeader("Accept", "*/*");
request.putHeader("Authorization", "Bearer " + TokenCache.getToken(cacheKey));
request.putHeader("App-Version", "3");
request.putHeader("platform", "web");
request.putHeader("Origin", shareOrigin);
request.putHeader("Referer", shareReferer);
request.putHeader("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36");
request.putHeader("Content-Type", "application/json;charset=UTF-8");
request.sendJsonObject(body).onSuccess(resp -> {
JsonObject json = asJson(resp);
if (json == null || json.getInteger("code", -1) != 0) {
promise.fail("v2接口返回异常: " + (json == null ? "null" : json.encode()));
return;
}
JsonObject data = json.getJsonObject("data", new JsonObject());
JsonArray dispatchList = data.getJsonArray("dispatchList", new JsonArray());
String downloadPath = data.getString("downloadPath");
if (StringUtils.isBlank(downloadPath)) {
promise.complete("");
return;
}
String prefix = "";
if (dispatchList.size() > 0) {
JsonObject firstDispatch = dispatchList.getJsonObject(0);
if (firstDispatch != null) {
prefix = firstDispatch.getString("prefix", "");
}
}
if (StringUtils.isBlank(prefix)) {
promise.complete(downloadPath);
return;
}
String finalUrl = prefix.endsWith("/") ? prefix.substring(0, prefix.length() - 1) : prefix;
finalUrl += downloadPath.startsWith("/") ? downloadPath : "/" + downloadPath;
promise.complete(finalUrl);
}).onFailure(promise::fail);
return promise.future();
}
private void getZipDownUrl(WebClient client, JsonObject fileInfo) {
JsonObject jsonObject = new JsonObject();
jsonObject.put("shareKey", YeShareHostUtil.normalizeShareKey(shareLinkInfo.getShareKey()));
jsonObject.put("fileIdList", new JsonArray().add(JsonObject.of("fileId", fileInfo.getInteger("FileId"))));
String timestamp = String.valueOf(System.currentTimeMillis());
String encryptedParams = encode123("/b/api/file/batch_download_share_info", "android", "55", timestamp);
String apiUrl = BATCH_DOWNLOAD_API_URL + encryptedParams;
log.info("Ye2 Batch Download API URL: {}", apiUrl);
HttpRequest<Buffer> bufferHttpRequest = client.postAbs(apiUrl);
bufferHttpRequest.putHeader("platform", "android");
bufferHttpRequest.putHeader("App-Version", "55");
bufferHttpRequest.putHeader("Authorization", "Bearer " + TokenCache.getToken(cacheKey));
bufferHttpRequest.putHeader("User-Agent", "Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36");
bufferHttpRequest.putHeader("Content-Type", "application/json");
bufferHttpRequest
.sendJsonObject(jsonObject)
.onSuccess(res2 -> handleDownloadUrlResponse(client, asJson(res2), "Ye2: 文件夹打包下载"))
.onFailure(err -> fail("文件夹打包下载接口失败: " + err.getMessage()));
}
private void handleDownloadUrlResponse(WebClient client, JsonObject downURLJson, String failPrefix) {
try { try {
if (downURLJson.getInteger("code") != 0) { if (downURLJson.getInteger("code") != 0) {
fail("Ye2: downURLJson返回值异常->" + downURLJson); fail(failPrefix + "返回值异常->" + downURLJson);
return; return;
} }
} catch (Exception ignored) { } catch (Exception ignored) {
fail("Ye2: downURLJson格式异常->" + downURLJson); fail(failPrefix + "格式异常->" + downURLJson);
return; return;
} }
@@ -342,7 +455,7 @@ public class Ye2Tool extends PanBase {
} }
if (StringUtils.isEmpty(downURL)) { if (StringUtils.isEmpty(downURL)) {
fail("Ye2: 未获取到下载链接"); fail(failPrefix + "未获取到下载链接");
return; return;
} }
@@ -350,7 +463,6 @@ public class Ye2Tool extends PanBase {
Map<String, String> urlParams = CommonUtils.getURLParams(downURL); Map<String, String> urlParams = CommonUtils.getURLParams(downURL);
String params = urlParams.get("params"); String params = urlParams.get("params");
if (StringUtils.isEmpty(params)) { if (StringUtils.isEmpty(params)) {
// 如果没有params参数直接使用downURL
complete(downURL); complete(downURL);
return; return;
} }
@@ -371,11 +483,11 @@ public class Ye2Tool extends PanBase {
JsonObject res3Json = asJson(res3); JsonObject res3Json = asJson(res3);
try { try {
if (res3Json.getInteger("code") != 0) { if (res3Json.getInteger("code") != 0) {
fail("Ye2: downUrl2返回值异常->" + res3Json); fail(failPrefix + "重定向返回值异常->" + res3Json);
return; return;
} }
} catch (Exception ignored) { } catch (Exception ignored) {
fail("Ye2: downUrl2格式异常->" + downURLJson); fail(failPrefix + "重定向格式异常->" + downURLJson);
return; return;
} }
String redirectUrl = res3Json.getJsonObject("data").getString("redirect_url"); String redirectUrl = res3Json.getJsonObject("data").getString("redirect_url");
@@ -386,112 +498,12 @@ public class Ye2Tool extends PanBase {
} }
}).onFailure(err -> fail("获取直链失败: " + err.getMessage())); }).onFailure(err -> fail("获取直链失败: " + err.getMessage()));
} catch (MalformedURLException e) { } catch (MalformedURLException e) {
// 如果解析失败直接使用downURL
complete(downURL); complete(downURL);
} catch (Exception e) { } catch (Exception e) {
fail("urlParams解析异常: " + e.getMessage()); fail("urlParams解析异常: " + e.getMessage());
} }
}).onFailure(err -> fail("下载接口失败: " + err.getMessage()));
} }
/**
* 获取文件夹打包下载链接使用Android平台API
*/
private void getZipDownUrl(WebClient client, JsonObject fileInfo) {
// 构建请求数据
JsonObject jsonObject = new JsonObject();
jsonObject.put("shareKey", shareLinkInfo.getShareKey().replace(".html", ""));
jsonObject.put("fileIdList", new JsonArray().add(JsonObject.of("fileId", fileInfo.getInteger("FileId"))));
// 使用encode123加密URL参数
String timestamp = String.valueOf(System.currentTimeMillis());
String encryptedParams = encode123("/b/api/file/batch_download_share_info", "android", "55", timestamp);
String apiUrl = BATCH_DOWNLOAD_API_URL + encryptedParams;
log.info("Ye2 Batch Download API URL: {}", apiUrl);
HttpRequest<Buffer> bufferHttpRequest = client.postAbs(apiUrl);
bufferHttpRequest.putHeader("platform", "android");
bufferHttpRequest.putHeader("App-Version", "55");
bufferHttpRequest.putHeader("Authorization", "Bearer " + ssoToken);
bufferHttpRequest.putHeader("User-Agent", "Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36");
bufferHttpRequest.putHeader("Content-Type", "application/json");
bufferHttpRequest
.sendJsonObject(jsonObject)
.onSuccess(res2 -> {
JsonObject downURLJson = asJson(res2);
try {
if (downURLJson.getInteger("code") != 0) {
fail("Ye2: 文件夹打包下载接口返回值异常->" + downURLJson);
return;
}
} catch (Exception ignored) {
fail("Ye2: 文件夹打包下载接口格式异常->" + downURLJson);
return;
}
String downURL = downURLJson.getJsonObject("data").getString("DownloadUrl");
if (StringUtils.isEmpty(downURL)) {
downURL = downURLJson.getJsonObject("data").getString("DownloadURL");
}
if (StringUtils.isEmpty(downURL)) {
fail("Ye2: 未获取到文件夹打包下载链接");
return;
}
try {
Map<String, String> urlParams = CommonUtils.getURLParams(downURL);
String params = urlParams.get("params");
if (StringUtils.isEmpty(params)) {
// 如果没有params参数直接使用downURL
complete(downURL);
return;
}
byte[] decodeByte = Base64.getDecoder().decode(params);
String downUrl2 = new String(decodeByte);
clientNoRedirects.getAbs(downUrl2).putHeaders(header).send().onSuccess(res3 -> {
if (res3.statusCode() == 302 || res3.statusCode() == 301) {
String redirectUrl = res3.getHeader("Location");
if (StringUtils.isBlank(redirectUrl)) {
fail("重定向链接为空");
return;
}
complete(redirectUrl);
return;
}
JsonObject res3Json = asJson(res3);
try {
if (res3Json.getInteger("code") != 0) {
fail("Ye2: 文件夹打包下载重定向返回值异常->" + res3Json);
return;
}
} catch (Exception ignored) {
fail("Ye2: 文件夹打包下载重定向格式异常->" + downURLJson);
return;
}
String redirectUrl = res3Json.getJsonObject("data").getString("redirect_url");
if (StringUtils.isNotEmpty(redirectUrl)) {
complete(redirectUrl);
} else {
complete(downUrl2);
}
}).onFailure(err -> fail("获取文件夹打包下载直链失败: " + err.getMessage()));
} catch (MalformedURLException e) {
// 如果解析失败直接使用downURL
complete(downURL);
} catch (Exception e) {
fail("文件夹打包下载urlParams解析异常: " + e.getMessage());
}
}).onFailure(err -> fail("文件夹打包下载接口失败: " + err.getMessage()));
}
/**
* 设置文件信息
*/
void setFileInfo(JsonObject reqBodyJson) { void setFileInfo(JsonObject reqBodyJson) {
FileInfo fileInfo = new FileInfo(); FileInfo fileInfo = new FileInfo();
fileInfo.setFileId(reqBodyJson.getInteger("FileId").toString()); fileInfo.setFileId(reqBodyJson.getInteger("FileId").toString());
@@ -514,62 +526,39 @@ public class Ye2Tool extends PanBase {
shareLinkInfo.getOtherParam().put("fileInfo", fileInfo); shareLinkInfo.getOtherParam().put("fileInfo", fileInfo);
} }
/**
* 解析文件夹中的文件列表
*/
@Override @Override
public Future<List<FileInfo>> parseFileList() { public Future<List<FileInfo>> parseFileList() {
Promise<List<FileInfo>> promise = Promise.promise(); Promise<List<FileInfo>> promise = Promise.promise();
String shareKey = shareLinkInfo.getShareKey().replace(".html", ""); String shareKey = YeShareHostUtil.normalizeShareKey(shareLinkInfo.getShareKey());
String pwd = shareLinkInfo.getSharePassword(); String pwd = shareLinkInfo.getSharePassword();
String parentFileId = "0"; // 根目录的文件ID String parentFileId = "0";
// 如果参数里的目录ID不为空则直接解析目录
String dirId = (String) shareLinkInfo.getOtherParam().get("dirId"); String dirId = (String) shareLinkInfo.getOtherParam().get("dirId");
if (StringUtils.isNotBlank(dirId)) { if (StringUtils.isNotBlank(dirId)) {
parentFileId = dirId; parentFileId = dirId;
} }
// 确保已登录 Future<String> tokenFuture = resolveTokenFuture();
Future<String> tokenFuture;
MultiMap auths = (MultiMap) shareLinkInfo.getOtherParam().get("auths");
if (auths != null && auths.contains("token")) {
String providedToken = auths.get("token");
if (StringUtils.isNotEmpty(providedToken)) {
ssoToken = providedToken;
tokenFuture = Future.succeededFuture(providedToken);
} else {
if (ssoToken == null || isTokenExpired()) {
tokenFuture = loginAndGetToken();
} else {
tokenFuture = Future.succeededFuture(ssoToken);
}
}
} else {
if (ssoToken == null || isTokenExpired()) {
tokenFuture = loginAndGetToken();
} else {
tokenFuture = Future.succeededFuture(ssoToken);
}
}
String finalParentFileId = parentFileId; String finalParentFileId = parentFileId;
tokenFuture.onSuccess(token -> { tokenFuture.onSuccess(token -> {
if (token.equals("nologin")) { if (token.equals("nologin")) {
promise.fail("该分享需要登录才能访问,请提供账号密码或token"); promise.fail("该分享需要登录才能访问,请配置认证信息");
return; return;
} }
// 构造文件列表接口的URL String normalizedShareKey = YeShareHostUtil.normalizeShareKey(shareKey);
String shareOrigin = resolveYeShareOrigin(normalizedShareKey);
String shareReferer = buildShareReferer(shareOrigin, normalizedShareKey, pwd);
client.getAbs(UriTemplate.of(GET_SHARE_INFO_URL)) client.getAbs(UriTemplate.of(GET_SHARE_INFO_URL))
.setTemplateParam("shareKey", shareKey) .setTemplateParam("shareKey", shareKey)
.setTemplateParam("pwd", StringUtils.isEmpty(pwd) ? "" : pwd) .setTemplateParam("pwd", StringUtils.isEmpty(pwd) ? "" : pwd)
.setTemplateParam("ParentFileId", finalParentFileId) .setTemplateParam("ParentFileId", finalParentFileId)
.putHeader("Authorization", "Bearer " + token) .putHeader("Authorization", "Bearer " + token)
.putHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36") .putHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36")
.putHeader("Referer", "https://www.123pan.com/") .putHeader("Referer", shareReferer)
.putHeader("Origin", "https://www.123pan.com") .putHeader("Origin", shareOrigin)
.send().onSuccess(res -> { .send().onSuccess(res -> {
JsonObject response = asJson(res); JsonObject response = asJson(res);
if (response.getInteger("code") != 0) { if (response.getInteger("code") != 0) {
@@ -585,13 +574,12 @@ public class Ye2Tool extends PanBase {
JsonArray infoList = response.getJsonObject("data").getJsonArray("InfoList"); JsonArray infoList = response.getJsonObject("data").getJsonArray("InfoList");
List<FileInfo> result = new ArrayList<>(); List<FileInfo> result = new ArrayList<>();
// 遍历返回的文件和目录信息
for (int i = 0; i < infoList.size(); i++) { for (int i = 0; i < infoList.size(); i++) {
JsonObject item = infoList.getJsonObject(i); JsonObject item = infoList.getJsonObject(i);
FileInfo fileInfo = new FileInfo(); FileInfo fileInfo = new FileInfo();
// 构建下载参数
JsonObject postData = JsonObject.of() JsonObject postData = JsonObject.of()
.put("shareKey", shareLinkInfo.getShareKey())
.put("driveId", 0) .put("driveId", 0)
.put("etag", item.getString("Etag")) .put("etag", item.getString("Etag"))
.put("fileId", item.getInteger("FileId")) .put("fileId", item.getInteger("FileId"))
@@ -602,7 +590,7 @@ public class Ye2Tool extends PanBase {
String param = CommonUtils.urlBase64Encode(postData.encode()); String param = CommonUtils.urlBase64Encode(postData.encode());
if (item.getInteger("Type") == 0) { // 文件 if (item.getInteger("Type") == 0) {
fileInfo.setFileName(item.getString("FileName")) fileInfo.setFileName(item.getString("FileName"))
.setFileId(item.getInteger("FileId").toString()) .setFileId(item.getInteger("FileId").toString())
.setFileType("file") .setFileType("file")
@@ -610,40 +598,20 @@ public class Ye2Tool extends PanBase {
.setHash(item.getString("Etag")) .setHash(item.getString("Etag"))
.setSizeStr(FileSizeConverter.convertToReadableSize(item.getLong("Size"))); .setSizeStr(FileSizeConverter.convertToReadableSize(item.getLong("Size")));
String createAt = item.getString("CreateAt"); setFileTimes(item, fileInfo);
if (StringUtils.isNotEmpty(createAt)) {
fileInfo.setCreateTime(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")
.format(OffsetDateTime.parse(createAt).toLocalDateTime()));
}
String updateAt = item.getString("UpdateAt");
if (StringUtils.isNotEmpty(updateAt)) {
fileInfo.setUpdateTime(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")
.format(OffsetDateTime.parse(updateAt).toLocalDateTime()));
}
fileInfo.setParserUrl(String.format("%s/v2/redirectUrl/%s/%s", getDomainName(), fileInfo.setParserUrl(String.format("%s/v2/redirectUrl/%s/%s", getDomainName(),
shareLinkInfo.getType(), param)) shareLinkInfo.getType(), param))
.setPreviewUrl(String.format("%s/v2/viewUrl/%s/%s", getDomainName(), .setPreviewUrl(String.format("%s/v2/viewUrl/%s/%s", getDomainName(),
shareLinkInfo.getType(), param)); shareLinkInfo.getType(), param));
result.add(fileInfo); result.add(fileInfo);
} else if (item.getInteger("Type") == 1) { // 目录 } else if (item.getInteger("Type") == 1) {
fileInfo.setFileName(item.getString("FileName")) fileInfo.setFileName(item.getString("FileName"))
.setFileId(item.getInteger("FileId").toString()) .setFileId(item.getInteger("FileId").toString())
.setFileType("folder") .setFileType("folder")
.setSize(0L); .setSize(0L);
String createAt = item.getString("CreateAt"); setFileTimes(item, fileInfo);
if (StringUtils.isNotEmpty(createAt)) {
fileInfo.setCreateTime(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")
.format(OffsetDateTime.parse(createAt).toLocalDateTime()));
}
String updateAt = item.getString("UpdateAt");
if (StringUtils.isNotEmpty(updateAt)) {
fileInfo.setUpdateTime(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")
.format(OffsetDateTime.parse(updateAt).toLocalDateTime()));
}
fileInfo.setParserUrl( fileInfo.setParserUrl(
String.format("%s/v2/getFileList?url=%s&dirId=%s&pwd=%s", String.format("%s/v2/getFileList?url=%s&dirId=%s&pwd=%s",
@@ -657,48 +625,70 @@ public class Ye2Tool extends PanBase {
} }
promise.complete(result); promise.complete(result);
}).onFailure(promise::fail); }).onFailure(promise::fail);
}).onFailure(err -> promise.fail("登录获取token失败: " + err.getMessage())); }).onFailure(err -> promise.fail("123盘解析异常: " + err.getMessage()));
return promise.future(); return promise.future();
} }
/**
* 通过ID解析特定文件
*/
@Override @Override
public Future<String> parseById() { public Future<String> parseById() {
JsonObject paramJson = (JsonObject) shareLinkInfo.getOtherParam().get("paramJson"); JsonObject paramJson = (JsonObject) shareLinkInfo.getOtherParam().get("paramJson");
Future<String> tokenFuture = resolveTokenFuture();
// 确保已登录
Future<String> tokenFuture;
MultiMap auths = (MultiMap) shareLinkInfo.getOtherParam().get("auths");
if (auths != null && auths.contains("token")) {
String providedToken = auths.get("token");
if (StringUtils.isNotEmpty(providedToken)) {
ssoToken = providedToken;
tokenFuture = Future.succeededFuture(providedToken);
} else {
if (ssoToken == null || isTokenExpired()) {
tokenFuture = loginAndGetToken();
} else {
tokenFuture = Future.succeededFuture(ssoToken);
}
}
} else {
if (ssoToken == null || isTokenExpired()) {
tokenFuture = loginAndGetToken();
} else {
tokenFuture = Future.succeededFuture(ssoToken);
}
}
tokenFuture.onSuccess(token -> { tokenFuture.onSuccess(token -> {
if (token.equals("nologin")) { if (token.equals("nologin")) {
fail("该分享需要登录才能下载,请提供账号密码或token"); fail("该分享需要登录才能下载,请配置认证信息");
return; return;
} }
// 使用encode123加密URL参数 String rawShareKey = paramJson.getString("shareKey");
if (StringUtils.isBlank(rawShareKey)) {
rawShareKey = paramJson.getString("ShareKey");
}
if (StringUtils.isBlank(rawShareKey)) {
rawShareKey = shareLinkInfo.getShareKey();
}
String normalizedShareKey = YeShareHostUtil.normalizeShareKey(rawShareKey);
if (StringUtils.isNotEmpty(normalizedShareKey)) {
JsonObject v2Body = new JsonObject()
.put("ShareKey", normalizedShareKey)
.put("FileID", paramJson.getInteger("fileId", paramJson.getInteger("FileID", 0)))
.put("S3keyFlag", paramJson.getString("s3keyFlag", paramJson.getString("S3keyFlag", "")))
.put("Size", paramJson.getLong("size", paramJson.getLong("Size", 0L)))
.put("Etag", paramJson.getString("etag", paramJson.getString("Etag", "")));
requestShareV2Download(normalizedShareKey, v2Body).onSuccess(v2Url -> {
if (StringUtils.isNotEmpty(v2Url)) {
complete(v2Url);
return;
}
parseByIdLegacy(token, paramJson);
}).onFailure(err -> {
log.warn("Ye2 parseById v2接口失败,回退旧接口: {}", err.getMessage());
parseByIdLegacy(token, paramJson);
});
return;
}
parseByIdLegacy(token, paramJson);
}).onFailure(err -> fail("123盘解析异常: " + err.getMessage()));
return promise.future();
}
private void parseByIdLegacy(String token, JsonObject paramJson) {
if (paramJson.containsKey("FileID") && !paramJson.containsKey("fileId")) {
paramJson.put("fileId", paramJson.getValue("FileID"));
}
if (paramJson.containsKey("S3keyFlag") && !paramJson.containsKey("s3keyFlag")) {
paramJson.put("s3keyFlag", paramJson.getValue("S3keyFlag"));
}
if (paramJson.containsKey("Size") && !paramJson.containsKey("size")) {
paramJson.put("size", paramJson.getValue("Size"));
}
if (paramJson.containsKey("Etag") && !paramJson.containsKey("etag")) {
paramJson.put("etag", paramJson.getValue("Etag"));
}
String timestamp = String.valueOf(System.currentTimeMillis()); String timestamp = String.valueOf(System.currentTimeMillis());
String encryptedParams = encode123("/b/api/file/download_info", "android", "55", timestamp); String encryptedParams = encode123("/b/api/file/download_info", "android", "55", timestamp);
String apiUrl = DOWNLOAD_API_URL + encryptedParams; String apiUrl = DOWNLOAD_API_URL + encryptedParams;
@@ -714,77 +704,57 @@ public class Ye2Tool extends PanBase {
bufferHttpRequest bufferHttpRequest
.sendJsonObject(paramJson) .sendJsonObject(paramJson)
.onSuccess(res2 -> { .onSuccess(res2 -> handleDownloadUrlResponse(client, asJson(res2), "Ye2"))
JsonObject downURLJson = asJson(res2); .onFailure(err -> fail("下载接口失败: " + err.getMessage()));
try {
if (downURLJson.getInteger("code") != 0) {
fail("Ye2: downURLJson返回值异常->" + downURLJson);
return;
}
} catch (Exception ignored) {
fail("Ye2: downURLJson格式异常->" + downURLJson);
return;
} }
String downURL = downURLJson.getJsonObject("data").getString("DownloadUrl"); private void setFileTimes(JsonObject item, FileInfo fileInfo) {
if (StringUtils.isEmpty(downURL)) { String createAt = item.getString("CreateAt");
downURL = downURLJson.getJsonObject("data").getString("DownloadURL"); if (StringUtils.isNotEmpty(createAt)) {
fileInfo.setCreateTime(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")
.format(OffsetDateTime.parse(createAt).toLocalDateTime()));
} }
if (StringUtils.isEmpty(downURL)) { String updateAt = item.getString("UpdateAt");
fail("Ye2: 未获取到下载链接"); if (StringUtils.isNotEmpty(updateAt)) {
return; fileInfo.setUpdateTime(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")
} .format(OffsetDateTime.parse(updateAt).toLocalDateTime()));
try {
Map<String, String> urlParams = CommonUtils.getURLParams(downURL);
String params = urlParams.get("params");
if (StringUtils.isEmpty(params)) {
// 如果没有params参数直接使用downURL
complete(downURL);
return;
}
byte[] decodeByte = Base64.getDecoder().decode(params);
String downUrl2 = new String(decodeByte);
clientNoRedirects.getAbs(downUrl2).putHeaders(header).send().onSuccess(res3 -> {
if (res3.statusCode() == 302 || res3.statusCode() == 301) {
String redirectUrl = res3.getHeader("Location");
if (StringUtils.isBlank(redirectUrl)) {
fail("重定向链接为空");
return;
}
complete(redirectUrl);
return;
}
JsonObject res3Json = asJson(res3);
try {
if (res3Json.getInteger("code") != 0) {
fail("Ye2: downUrl2返回值异常->" + res3Json);
return;
}
} catch (Exception ignored) {
fail("Ye2: downUrl2格式异常->" + downURLJson);
return;
}
String redirectUrl = res3Json.getJsonObject("data").getString("redirect_url");
if (StringUtils.isNotEmpty(redirectUrl)) {
complete(redirectUrl);
} else {
complete(downUrl2);
}
}).onFailure(err -> fail("获取直链失败: " + err.getMessage()));
} catch (MalformedURLException e) {
// 如果解析失败直接使用downURL
complete(downURL);
} catch (Exception e) {
fail("urlParams解析异常: " + e.getMessage());
}
}).onFailure(err -> fail("下载接口失败: " + err.getMessage()));
}).onFailure(err -> fail("登录获取token失败: " + err.getMessage()));
return promise.future();
} }
} }
private String resolveYeShareOrigin(String shareKey) {
String origin = extractYeShareOrigin(shareLinkInfo.getShareUrl());
if (StringUtils.isNotBlank(origin)) {
return origin;
}
origin = extractYeShareOrigin(shareLinkInfo.getStandardUrl());
if (StringUtils.isNotBlank(origin)) {
return origin;
}
String uid = YeShareHostUtil.getNumericSubdomainIdByShareKey(shareKey);
if (StringUtils.isNotBlank(uid)) {
return "https://" + uid + ".share.123pan.cn";
}
return "https://www.123pan.com";
}
private String extractYeShareOrigin(String url) {
if (StringUtils.isBlank(url) || !url.matches("^https?://[a-zA-Z\\d-]+\\.(?:mshare|share)\\.123pan\\.cn(?:[:/].*)?$")) {
return "";
}
int idx = url.indexOf('/', url.indexOf("//") + 2);
return idx > 0 ? url.substring(0, idx) : url;
}
private String buildShareReferer(String shareOrigin, String shareKey, String pwd) {
String key = YeShareHostUtil.normalizeShareKey(shareKey);
if (StringUtils.isBlank(key)) {
return shareOrigin + "/";
}
String referer = shareOrigin + "/123pan/" + key;
if (StringUtils.isNotBlank(pwd)) {
referer += "?pwd=" + pwd;
}
return referer;
}
}
@@ -0,0 +1,694 @@
package cn.qaiu.parser.impl;
import cn.qaiu.entity.FileInfo;
import cn.qaiu.entity.ShareLinkInfo;
import cn.qaiu.parser.PanBase;
import cn.qaiu.util.CommonUtils;
import cn.qaiu.util.FileSizeConverter;
import io.vertx.core.Future;
import io.vertx.core.Promise;
import io.vertx.core.buffer.Buffer;
import io.vertx.core.json.JsonArray;
import io.vertx.core.json.JsonObject;
import io.vertx.ext.web.client.HttpRequest;
import io.vertx.ext.web.client.HttpResponse;
import org.apache.commons.lang3.StringUtils;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* 永硕E盘 ysepan.com / ys168.com cccpan.com / ysupan.com / uupan.net / ysok.net
* <p>
* 空间分享形如 https://{space}.ysepan.com/ 需空间访问密码时通过 sharePassword 传入
*/
public class YsTool extends PanBase {
private static final String BROWSER_UA =
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
+ "(KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36";
private static final Pattern ANTIFORGERY_PATTERN =
Pattern.compile("RequestVerificationToken'\\s*:\\s*'([^']+)'");
private static final Pattern HTXX_PATTERN =
Pattern.compile("window\\.htxx\\s*=\\s*(\\{.*?});", Pattern.DOTALL);
private static final Pattern JWT_COOKIE_PATTERN =
Pattern.compile("jwttk_[^=]+=([^;\\s]+)");
private static final String PARAM_DIR_ID = "dirId";
/** 永硕目录内的子目录名(API 字段 zml),用于二级层级 */
private static final String PARAM_ZML = "zml";
public YsTool(ShareLinkInfo shareLinkInfo) {
super(shareLinkInfo);
}
@Override
public Future<String> parse() {
ensureSession().onSuccess(session -> {
Object dirIdObj = shareLinkInfo.getOtherParam().get(PARAM_DIR_ID);
if (dirIdObj != null && StringUtils.isNotBlank(dirIdObj.toString())) {
int dirId = Integer.parseInt(dirIdObj.toString());
fetchFiles(session, dirId).onSuccess(filesResp -> {
List<JsonObject> files = downloadableFiles(filesResp);
if (files.isEmpty()) {
fail("目录内没有可下载文件");
return;
}
completeDownload(session, filesResp, files.get(0));
}).onFailure(err -> fail(err, err.getMessage()));
return;
}
fetchDirectories(session).compose(dirs -> collectDownloadableFiles(session, dirs))
.onSuccess(collected -> {
if (collected.isEmpty()) {
fail("空间内没有可下载文件,请检查密码或目录权限");
return;
}
if (collected.size() > 1) {
fail("空间包含多个文件(共{}个),请使用文件列表接口后再按文件解析", collected.size());
return;
}
CollectedFile only = collected.get(0);
completeDownload(session, only.filesResp, only.file);
}).onFailure(err -> fail(err, err.getMessage()));
}).onFailure(err -> fail(err, err.getMessage()));
return promise.future();
}
@Override
public Future<List<FileInfo>> parseFileList() {
Promise<List<FileInfo>> listPromise = Promise.promise();
ensureSession().onSuccess(session -> {
Object dirIdObj = shareLinkInfo.getOtherParam().get(PARAM_DIR_ID);
if (dirIdObj != null && StringUtils.isNotBlank(dirIdObj.toString())) {
int dirId = Integer.parseInt(dirIdObj.toString());
String zmlFilter = currentZmlFilter();
fetchFiles(session, dirId).onSuccess(filesResp -> {
try {
listPromise.complete(mapFiles(session, dirId, filesResp, zmlFilter));
} catch (Exception e) {
listPromise.fail(baseMsg() + " - 解析文件列表失败: " + e.getMessage());
}
}).onFailure(listPromise::fail);
return;
}
fetchDirectories(session).onSuccess(dirs -> {
try {
listPromise.complete(mapDirectories(session, dirs));
} catch (Exception e) {
listPromise.fail(baseMsg() + " - 解析目录列表失败: " + e.getMessage());
}
}).onFailure(listPromise::fail);
}).onFailure(listPromise::fail);
return listPromise.future();
}
@Override
public Future<String> parseById() {
JsonObject paramJson = (JsonObject) shareLinkInfo.getOtherParam().get("paramJson");
if (paramJson == null) {
fail("缺少 paramJson 参数");
return promise.future();
}
String downloadUrl = paramJson.getString("downloadUrl");
if (StringUtils.isNotBlank(downloadUrl)) {
completeWithMeta(downloadUrl, downloadHeaders(paramJson.getString("referer")));
return promise.future();
}
String space = paramJson.getString("space", spaceName());
String xzpz = paramJson.getString("xzpz");
String pz = paramJson.getString("pz");
String fwq = paramJson.getString("fwq");
String fileName = paramJson.getString("fileName");
if (StringUtils.isAnyBlank(space, xzpz, pz, fwq, fileName)) {
fail("下载参数不完整: {}", paramJson);
return promise.future();
}
String url = buildDownloadUrl(space, xzpz, pz, fwq, fileName);
completeWithMeta(url, downloadHeaders(paramJson.getString("referer", spaceOrigin())));
return promise.future();
}
private Future<Session> ensureSession() {
Promise<Session> p = Promise.promise();
String space = spaceName();
if (StringUtils.isBlank(space)) {
p.fail(baseMsg() + " - 空间名为空");
return p.future();
}
String origin = spaceOrigin();
clientSession.getAbs(origin + "/")
.putHeader("User-Agent", BROWSER_UA)
.putHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
.send()
.compose(res -> handleSpaceHome(origin, space, res))
.onSuccess(p::complete)
.onFailure(p::fail);
return p.future();
}
private Future<Session> handleSpaceHome(String origin, String space, HttpResponse<Buffer> res) {
String html = res.bodyAsString() != null ? res.bodyAsString() : "";
String jwt = extractJwt(res);
JsonObject htxx = extractHtxx(html);
// 已进入空间
if (htxx != null && StringUtils.isNotBlank(htxx.getString("qqdz"))) {
if (StringUtils.isBlank(jwt)) {
return Future.failedFuture(baseMsg() + " - 无法获取会话令牌");
}
return Future.succeededFuture(toSession(origin, space, jwt, htxx));
}
// 需要空间密码
if (html.contains("VerifyPassword") || html.contains("loginPopup")) {
String pwd = shareLinkInfo.getSharePassword();
if (StringUtils.isBlank(pwd)) {
return Future.failedFuture(baseMsg() + " - 空间需要访问密码");
}
String antiforgery = extractAntiforgery(html);
if (StringUtils.isBlank(antiforgery)) {
return Future.failedFuture(baseMsg() + " - 无法获取防伪令牌");
}
return verifyPassword(origin, space, pwd, antiforgery)
.compose(verifiedJwt -> reloadSpace(origin, space, verifiedJwt));
}
return Future.failedFuture(baseMsg() + " - 无法解析空间信息");
}
private Future<String> verifyPassword(String origin, String space, String pwd, String antiforgery) {
Promise<String> p = Promise.promise();
JsonObject body = new JsonObject()
.put("password", pwd)
.put("dlmc", space)
.put("remember", false);
clientSession.postAbs(origin + "/?handler=VerifyPassword")
.putHeader("User-Agent", BROWSER_UA)
.putHeader("Content-Type", "application/json")
.putHeader("Origin", origin)
.putHeader("Referer", origin + "/")
.putHeader("RequestVerificationToken", antiforgery)
.sendJsonObject(body)
.onSuccess(res -> {
try {
JsonObject json = res.bodyAsJsonObject();
if (json == null || !Boolean.TRUE.equals(json.getBoolean("success"))) {
String msg = json != null ? json.getString("message", "密码错误") : "密码验证失败";
p.fail(baseMsg() + " - " + msg);
return;
}
String jwt = extractJwt(res);
if (StringUtils.isBlank(jwt)) {
p.fail(baseMsg() + " - 密码验证成功但未返回会话令牌");
return;
}
p.complete(jwt);
} catch (Exception e) {
p.fail(baseMsg() + " - 密码验证响应异常: " + e.getMessage());
}
})
.onFailure(t -> p.fail(baseMsg() + " - 密码验证请求失败: " + t.getMessage()));
return p.future();
}
private Future<Session> reloadSpace(String origin, String space, String verifiedJwt) {
Promise<Session> p = Promise.promise();
clientSession.getAbs(origin + "/")
.putHeader("User-Agent", BROWSER_UA)
.putHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
.send()
.onSuccess(res -> {
String html = res.bodyAsString() != null ? res.bodyAsString() : "";
JsonObject htxx = extractHtxx(html);
String jwt = StringUtils.defaultIfBlank(extractJwt(res), verifiedJwt);
if (htxx == null || StringUtils.isBlank(htxx.getString("qqdz"))) {
p.fail(baseMsg() + " - 密码验证后仍无法进入空间");
return;
}
if (StringUtils.isBlank(jwt)) {
p.fail(baseMsg() + " - 密码验证后无法获取会话令牌");
return;
}
p.complete(toSession(origin, space, jwt, htxx));
})
.onFailure(t -> p.fail(baseMsg() + " - 重新加载空间失败: " + t.getMessage()));
return p.future();
}
private Session toSession(String origin, String space, String jwt, JsonObject htxx) {
String apiBase = htxx.getString("qqdz");
if (apiBase != null && !apiBase.endsWith("/")) {
apiBase = apiBase + "/";
}
String dlmc = htxx.getString("dlmc", space);
return new Session(origin, dlmc, jwt, apiBase);
}
private Future<JsonArray> fetchDirectories(Session session) {
Promise<JsonArray> p = Promise.promise();
apiPost(session, "ml/mldq", new JsonObject())
.onSuccess(json -> {
JsonArray lb = json.getJsonArray("lb");
p.complete(lb != null ? lb : new JsonArray());
})
.onFailure(p::fail);
return p.future();
}
private Future<JsonObject> fetchFiles(Session session, int dirId) {
Promise<JsonObject> p = Promise.promise();
JsonObject body = new JsonObject()
.put("mlbh", dirId)
.put("kqmm", "")
.put("wjbh", 0)
.put("ip1", "");
apiPost(session, "wj/wjdq", body)
.onSuccess(p::complete)
.onFailure(p::fail);
return p.future();
}
private Future<JsonObject> apiPost(Session session, String path, JsonObject body) {
Promise<JsonObject> p = Promise.promise();
String url = session.apiBase + path;
HttpRequest<Buffer> req = clientSession.postAbs(url)
.putHeader("User-Agent", BROWSER_UA)
.putHeader("Accept", "application/json, text/plain, */*")
.putHeader("Content-Type", "application/json")
.putHeader("Origin", session.origin)
.putHeader("Referer", session.origin + "/")
.putHeader("Authorization", "Bearer " + session.jwt);
req.sendJsonObject(body)
.onSuccess(res -> {
try {
if (res.statusCode() >= 400) {
p.fail(baseMsg() + " - API " + path + " HTTP " + res.statusCode());
return;
}
JsonObject json = asJson(res);
if (json == null || json.isEmpty()) {
p.fail(baseMsg() + " - API " + path + " 返回空响应");
return;
}
p.complete(json);
} catch (Exception e) {
p.fail(baseMsg() + " - API " + path + " 响应异常: " + e.getMessage());
}
})
.onFailure(t -> p.fail(baseMsg() + " - API " + path + " 请求失败: " + t.getMessage()));
return p.future();
}
private Future<List<CollectedFile>> collectDownloadableFiles(Session session, JsonArray dirs) {
Promise<List<CollectedFile>> p = Promise.promise();
List<CollectedFile> collected = new ArrayList<>();
List<Integer> dirIds = new ArrayList<>();
for (int i = 0; i < dirs.size(); i++) {
JsonObject dir = dirs.getJsonObject(i);
if (isAccessibleDirectory(dir)) {
dirIds.add(dir.getInteger("bh"));
}
}
if (dirIds.isEmpty()) {
p.complete(collected);
return p.future();
}
fetchNextDirFiles(session, dirIds, 0, collected, p);
return p.future();
}
private void fetchNextDirFiles(Session session, List<Integer> dirIds, int index,
List<CollectedFile> collected, Promise<List<CollectedFile>> promise) {
if (index >= dirIds.size()) {
promise.complete(collected);
return;
}
int dirId = dirIds.get(index);
fetchFiles(session, dirId).onSuccess(filesResp -> {
for (JsonObject file : downloadableFiles(filesResp)) {
collected.add(new CollectedFile(filesResp, file));
}
fetchNextDirFiles(session, dirIds, index + 1, collected, promise);
}).onFailure(promise::fail);
}
private List<FileInfo> mapDirectories(Session session, JsonArray dirs) {
List<FileInfo> result = new ArrayList<>();
for (int i = 0; i < dirs.size(); i++) {
JsonObject dir = dirs.getJsonObject(i);
if (!isAccessibleDirectory(dir)) {
continue;
}
Integer bh = dir.getInteger("bh");
if (bh == null) {
continue;
}
String title = StringUtils.defaultIfBlank(dir.getString("bt"), "目录" + bh);
FileInfo info = new FileInfo()
.setFileName(title)
.setFileId(bh.toString())
.setFileType("folder")
.setSize(0L)
.setSizeStr("0B")
.setDescription(dir.getString("sm", ""))
.setCreateTime(normalizeTime(dir.getString("sj")))
.setPanType(shareLinkInfo.getType())
.setParserUrl(String.format("%s/v2/getFileList?url=%s&dirId=%s&pwd=%s",
getDomainName(),
urlEncode(shareLinkInfo.getShareUrl()),
bh,
urlEncode(StringUtils.defaultString(shareLinkInfo.getSharePassword()))));
result.add(info);
}
return result;
}
/**
* 将目录内文件按 zml子目录分层
* <ul>
* <li>zmlFilter 为空返回子目录folder+ 根级文件</li>
* <li>zmlFilter 非空仅返回该子目录下的文件/链接</li>
* </ul>
*/
private List<FileInfo> mapFiles(Session session, int dirId, JsonObject filesResp, String zmlFilter) {
List<FileInfo> result = new ArrayList<>();
String xzpz = filesResp.getJsonObject("ml", new JsonObject()).getString("xzpz", "");
JsonArray lb = filesResp.getJsonArray("lb", new JsonArray());
boolean listingSubdir = StringUtils.isNotBlank(zmlFilter);
// 未进入子目录时先按出现顺序收集 zml 作为二级文件夹
if (!listingSubdir) {
java.util.LinkedHashSet<String> subdirs = new java.util.LinkedHashSet<>();
for (int i = 0; i < lb.size(); i++) {
JsonObject item = lb.getJsonObject(i);
if (item == null) {
continue;
}
String zml = StringUtils.defaultString(item.getString("zml")).trim();
if (StringUtils.isNotBlank(zml)) {
subdirs.add(zml);
}
}
for (String zml : subdirs) {
result.add(new FileInfo()
.setFileName(zml)
.setFileId(dirId + ":" + zml)
.setFileType("folder")
.setSize(0L)
.setSizeStr("0B")
.setFilePath(zml)
.setPanType(shareLinkInfo.getType())
.setParserUrl(String.format("%s/v2/getFileList?url=%s&dirId=%s&zml=%s&pwd=%s",
getDomainName(),
urlEncode(shareLinkInfo.getShareUrl()),
dirId,
urlEncode(zml),
urlEncode(StringUtils.defaultString(shareLinkInfo.getSharePassword())))));
}
}
for (int i = 0; i < lb.size(); i++) {
JsonObject item = lb.getJsonObject(i);
if (item == null) {
continue;
}
String itemZml = StringUtils.defaultString(item.getString("zml")).trim();
if (listingSubdir) {
if (!zmlFilter.equals(itemZml)) {
continue;
}
} else if (StringUtils.isNotBlank(itemZml)) {
// 根级列表只展示无 zml 的条目 zml 的归入子目录
continue;
}
Integer bh = item.getInteger("bh");
if (bh == null) {
continue;
}
String wjlx = item.getString("wjlx", "");
// URL / 公告条目
if ("url".equalsIgnoreCase(wjlx)) {
String link = StringUtils.defaultString(item.getString("wjm")).trim();
String title = StringUtils.defaultString(item.getString("bt")).trim();
// 空占位标题和链接都空跳过与官网展示一致
if (StringUtils.isAllBlank(title, link)) {
continue;
}
if (StringUtils.isBlank(title)) {
title = StringUtils.defaultIfBlank(link, "链接");
}
FileInfo urlInfo = new FileInfo()
.setFileName(title)
.setFileId(bh.toString())
.setFileType("url")
.setSize(0L)
.setSizeStr("0B")
.setFilePath(itemZml)
.setCreateTime(normalizeTime(item.getString("sj")))
.setPanType(shareLinkInfo.getType())
.setPreviewUrl(link)
.setDescription(link);
// parserUrl 置空避免前端误走下载打开走 previewUrl
result.add(urlInfo);
continue;
}
String fileName = item.getString("wjm");
String fwq = item.getString("fwq");
String pz = item.getString("pz");
if (StringUtils.isAnyBlank(fileName, fwq, pz, xzpz)) {
continue;
}
long size = item.getLong("dx", 0L);
String downloadUrl = buildDownloadUrl(session.space, xzpz, pz, fwq, fileName);
JsonObject param = new JsonObject()
.put("space", session.space)
.put("xzpz", xzpz)
.put("pz", pz)
.put("fwq", fwq)
.put("fileName", fileName)
.put("mlbh", dirId)
.put("fileId", bh)
.put("downloadUrl", downloadUrl)
.put("referer", session.origin + "/");
String paramEncoded = CommonUtils.urlBase64Encode(param.encode());
FileInfo fileInfo = new FileInfo()
.setFileName(fileName)
.setFileId(bh.toString())
.setFileType("file")
.setSize(size)
.setSizeStr(FileSizeConverter.convertToReadableSize(size))
.setFilePath(itemZml)
.setCreateTime(normalizeTime(item.getString("sj")))
.setPanType(shareLinkInfo.getType())
.setParserUrl(String.format("%s/v2/redirectUrl/%s/%s",
getDomainName(), shareLinkInfo.getType(), paramEncoded))
.setPreviewUrl(String.format("%s/v2/viewUrl/%s/%s",
getDomainName(), shareLinkInfo.getType(), paramEncoded));
result.add(fileInfo);
}
return result;
}
private String currentZmlFilter() {
Object zml = shareLinkInfo.getOtherParam().get(PARAM_ZML);
if (zml == null) {
return "";
}
try {
return java.net.URLDecoder.decode(zml.toString(), StandardCharsets.UTF_8).trim();
} catch (Exception e) {
return zml.toString().trim();
}
}
private List<JsonObject> downloadableFiles(JsonObject filesResp) {
List<JsonObject> files = new ArrayList<>();
String xzpz = filesResp.getJsonObject("ml", new JsonObject()).getString("xzpz", "");
if (StringUtils.isBlank(xzpz)) {
return files;
}
JsonArray lb = filesResp.getJsonArray("lb", new JsonArray());
for (int i = 0; i < lb.size(); i++) {
JsonObject item = lb.getJsonObject(i);
if (item == null) {
continue;
}
if ("url".equalsIgnoreCase(item.getString("wjlx", ""))) {
continue;
}
if (StringUtils.isAnyBlank(item.getString("wjm"), item.getString("fwq"), item.getString("pz"))) {
continue;
}
files.add(item);
}
return files;
}
private void completeDownload(Session session, JsonObject filesResp, JsonObject file) {
String xzpz = filesResp.getJsonObject("ml", new JsonObject()).getString("xzpz");
String url = buildDownloadUrl(session.space, xzpz, file.getString("pz"),
file.getString("fwq"), file.getString("wjm"));
FileInfo fileInfo = new FileInfo()
.setFileName(file.getString("wjm"))
.setFileId(String.valueOf(file.getInteger("bh")))
.setSize(file.getLong("dx", 0L))
.setSizeStr(FileSizeConverter.convertToReadableSize(file.getLong("dx", 0L)))
.setFileType("file")
.setFilePath(file.getString("zml", ""))
.setCreateTime(normalizeTime(file.getString("sj")))
.setPanType(shareLinkInfo.getType());
shareLinkInfo.getOtherParam().put("fileInfo", fileInfo);
completeWithMeta(url, downloadHeaders(session.origin + "/"));
}
/**
* 拼装直链注意不要在 xzpz 前加 "_"官方页面直链无此前缀加了会 404
*/
static String buildDownloadUrl(String space, String xzpz, String pz, String fwq, String fileName) {
String host = "X".equalsIgnoreCase(fwq)
? "y.ys168.com:8000"
: "ys-" + fwq.toLowerCase() + ".ysepan.com";
return "https://" + host + "/wap/"
+ encodePathSegment(space) + "/"
+ encodePathSegment(xzpz) + "/"
+ encodePathSegment(pz) + "/"
+ encodePathSegment(fileName);
}
private static String encodePathSegment(String value) {
// 保留永硕 token 中常见的 . , _ - 等字符仅编码必须转义的部分
return URLEncoder.encode(value, StandardCharsets.UTF_8)
.replace("+", "%20")
.replace("%2E", ".")
.replace("%2C", ",")
.replace("%5F", "_")
.replace("%2D", "-");
}
private Map<String, String> downloadHeaders(String referer) {
Map<String, String> headers = new HashMap<>();
headers.put("User-Agent", BROWSER_UA);
if (StringUtils.isNotBlank(referer)) {
headers.put("Referer", referer);
}
return headers;
}
private boolean isAccessibleDirectory(JsonObject dir) {
if (dir == null) {
return false;
}
// 无可下载权限或需特殊打开方式的目录忽略
if (!Boolean.TRUE.equals(dir.getBoolean("qxz", true))) {
return false;
}
Integer kqfs = dir.getInteger("kqfs", 0);
return kqfs == null || kqfs == 0;
}
private String spaceName() {
String key = shareLinkInfo.getShareKey();
if (StringUtils.isBlank(key)) {
return null;
}
try {
return java.net.URLDecoder.decode(key, StandardCharsets.UTF_8);
} catch (Exception e) {
return key;
}
}
private String spaceOrigin() {
String shareUrl = shareLinkInfo.getShareUrl();
if (StringUtils.isNotBlank(shareUrl)) {
try {
java.net.URI uri = java.net.URI.create(shareUrl);
String scheme = uri.getScheme() != null ? uri.getScheme() : "https";
return scheme + "://" + uri.getHost();
} catch (Exception ignored) {
}
}
return "https://" + spaceName() + ".ysepan.com";
}
private static String extractAntiforgery(String html) {
Matcher m = ANTIFORGERY_PATTERN.matcher(html);
return m.find() ? m.group(1) : null;
}
private static JsonObject extractHtxx(String html) {
Matcher m = HTXX_PATTERN.matcher(html);
if (!m.find()) {
return null;
}
try {
return new JsonObject(m.group(1));
} catch (Exception e) {
return null;
}
}
private static String extractJwt(HttpResponse<Buffer> res) {
List<String> setCookies = res.cookies();
if (setCookies != null) {
for (String c : setCookies) {
Matcher m = JWT_COOKIE_PATTERN.matcher(c);
if (m.find()) {
return m.group(1);
}
}
}
String raw = res.getHeader("Set-Cookie");
if (raw != null) {
Matcher m = JWT_COOKIE_PATTERN.matcher(raw);
if (m.find()) {
return m.group(1);
}
}
return null;
}
private static String normalizeTime(String sj) {
if (StringUtils.isBlank(sj)) {
return sj;
}
// 2024-10-26T10:12:54.98 -> 2024-10-26 10:12:54
String t = sj.replace('T', ' ');
int dot = t.indexOf('.');
if (dot > 0) {
t = t.substring(0, dot);
}
return t;
}
private static String urlEncode(String value) {
return URLEncoder.encode(StringUtils.defaultString(value), StandardCharsets.UTF_8);
}
private record Session(String origin, String space, String jwt, String apiBase) {
}
private record CollectedFile(JsonObject filesResp, JsonObject file) {
}
}
@@ -14,7 +14,6 @@ import java.security.spec.X509EncodedKeySpec;
import java.util.Base64; import java.util.Base64;
import java.util.Date; import java.util.Date;
import java.util.HexFormat; import java.util.HexFormat;
import java.util.Random;
import java.util.regex.Matcher; import java.util.regex.Matcher;
import java.util.regex.Pattern; import java.util.regex.Pattern;
@@ -299,7 +298,7 @@ public class AESUtils {
//length用户要求产生字符串的长度 //length用户要求产生字符串的长度
public static String getRandomString(int length){ public static String getRandomString(int length){
String str="abcdefghijklmnopqrstuvwxyz0123456789"; String str="abcdefghijklmnopqrstuvwxyz0123456789";
Random random=new Random(); SecureRandom random=new SecureRandom();
StringBuilder sb=new StringBuilder(); StringBuilder sb=new StringBuilder();
for(int i=0;i<length;i++){ for(int i=0;i<length;i++){
int number=random.nextInt(36); int number=random.nextInt(36);
@@ -33,6 +33,9 @@ public class CommonUtils {
public static Map<String, String> getURLParams(String url) throws MalformedURLException { public static Map<String, String> getURLParams(String url) throws MalformedURLException {
URL fullUrl = new URL(url); URL fullUrl = new URL(url);
String query = fullUrl.getQuery(); String query = fullUrl.getQuery();
if (query == null || query.isEmpty()) {
return new HashMap<>();
}
String[] params = query.split("&"); String[] params = query.split("&");
Map<String, String> map = new HashMap<>(); Map<String, String> map = new HashMap<>();
for (String param : params) { for (String param : params) {
@@ -74,32 +77,57 @@ public class CommonUtils {
} }
/** /**
* urlEncode -> deBase64 -> string * 解码路径参数中的 Base64
* @param encoded 编码后的字符串 * <p>优先按 URL-Safe Base64 兼容历史标准 Base64 + URLEncode以及重复 encode</p>
* @return 解码后的字符串
*/ */
public static String urlBase64Decode(String encoded) { public static String urlBase64Decode(String encoded) {
try { if (encoded == null || encoded.isEmpty()) {
String urlDecoded = java.net.URLDecoder.decode(encoded, StandardCharsets.UTF_8); throw new RuntimeException("URL Base64 解码失败: empty");
byte[] base64DecodedBytes = java.util.Base64.getDecoder().decode(urlDecoded);
return new String(base64DecodedBytes, java.nio.charset.StandardCharsets.UTF_8);
} catch (Exception e) {
throw new RuntimeException("URL Base64 解码失败", e);
} }
String s = encoded.trim().replace(' ', '+');
// 兼容历史 URLEncode / 误二次 encode % 则解到不再变化
for (int i = 0; i < 3 && s.contains("%"); i++) {
try {
String next = java.net.URLDecoder.decode(s, StandardCharsets.UTF_8);
if (next.equals(s)) {
break;
}
s = next;
} catch (Exception e) {
break;
}
}
Exception last = null;
for (String candidate : new String[]{s, padBase64(s)}) {
try {
return new String(java.util.Base64.getUrlDecoder().decode(candidate), StandardCharsets.UTF_8);
} catch (Exception e) {
last = e;
}
try {
return new String(java.util.Base64.getDecoder().decode(candidate), StandardCharsets.UTF_8);
} catch (Exception e) {
last = e;
}
}
throw new RuntimeException("URL Base64 解码失败", last);
} }
/** /**
* string -> base64Encode -> urlEncode * 编码为可直接放进 URL path Base64URL-Safe padding
* @param str 原始字符串 * <p>不再做 URLEncoder避免前端/代理再 encode 时变成 %253D</p>
* @return 编码后的字符串
*/ */
public static String urlBase64Encode(String str) { public static String urlBase64Encode(String str) {
try { return java.util.Base64.getUrlEncoder()
byte[] base64EncodedBytes = java.util.Base64.getEncoder().encode(str.getBytes(java.nio.charset.StandardCharsets.UTF_8)); .withoutPadding()
String base64Encoded = new String(base64EncodedBytes, java.nio.charset.StandardCharsets.UTF_8); .encodeToString(str.getBytes(StandardCharsets.UTF_8));
return java.net.URLEncoder.encode(base64Encoded, StandardCharsets.UTF_8);
} catch (Exception e) {
throw new RuntimeException("URL Base64 编码失败", e);
} }
private static String padBase64(String s) {
int mod = s.length() % 4;
if (mod == 0) {
return s;
}
return s + "====".substring(mod);
} }
} }
@@ -17,16 +17,35 @@ import java.util.zip.InflaterInputStream;
public class HttpResponseHelper { public class HttpResponseHelper {
static Logger LOGGER = LoggerFactory.getLogger(HttpResponseHelper.class); static Logger LOGGER = LoggerFactory.getLogger(HttpResponseHelper.class);
private static final int MAX_RESPONSE_BODY_BYTES = 8 * 1024 * 1024;
private static final int MAX_DECOMPRESSED_CHARS = 16 * 1024 * 1024;
// -------------------- 公共方法 -------------------- // -------------------- 公共方法 --------------------
public static String asText(HttpResponse<?> res) { public static String asText(HttpResponse<?> res) {
String encoding = res.getHeader(HttpHeaders.CONTENT_ENCODING.toString()); String encoding = res.getHeader(HttpHeaders.CONTENT_ENCODING.toString());
try { try {
Buffer body = toBuffer(res); Buffer body = toBuffer(res);
return asText(body, encoding);
} catch (IllegalArgumentException | UnsupportedOperationException e) {
throw e;
} catch (Exception e) {
LOGGER.error("asText: {}", e.getMessage(), e);
return null;
}
}
public static String asText(Buffer body, String encoding) {
try {
if (body == null) {
return "";
}
ensureBodyLimit(body);
if (encoding == null || "identity".equalsIgnoreCase(encoding)) { if (encoding == null || "identity".equalsIgnoreCase(encoding)) {
return body.toString(StandardCharsets.UTF_8); return body.toString(StandardCharsets.UTF_8);
} }
return decompress(body, encoding); return decompress(body, encoding);
} catch (IllegalArgumentException | UnsupportedOperationException e) {
throw e;
} catch (Exception e) { } catch (Exception e) {
LOGGER.error("asText: {}", e.getMessage(), e); LOGGER.error("asText: {}", e.getMessage(), e);
return null; return null;
@@ -36,6 +55,29 @@ public class HttpResponseHelper {
public static JsonObject asJson(HttpResponse<?> res) { public static JsonObject asJson(HttpResponse<?> res) {
try { try {
String text = asText(res); String text = asText(res);
return parseJsonText(text);
} catch (IllegalArgumentException | UnsupportedOperationException e) {
throw e;
} catch (Exception e) {
LOGGER.error("asJson: {}", e.getMessage(), e);
return JsonObject.of();
}
}
public static JsonObject asJson(Buffer body, String encoding) {
try {
String text = asText(body, encoding);
return parseJsonText(text);
} catch (IllegalArgumentException | UnsupportedOperationException e) {
throw e;
} catch (Exception e) {
LOGGER.error("asJson: {}", e.getMessage(), e);
return JsonObject.of();
}
}
private static JsonObject parseJsonText(String text) {
try {
if (text != null) { if (text != null) {
return new JsonObject(text); return new JsonObject(text);
} else { } else {
@@ -53,13 +95,26 @@ public class HttpResponseHelper {
return res.body() instanceof Buffer ? (Buffer) res.body() : Buffer.buffer(res.bodyAsString()); return res.body() instanceof Buffer ? (Buffer) res.body() : Buffer.buffer(res.bodyAsString());
} }
private static void ensureBodyLimit(Buffer body) {
if (body != null && body.length() > MAX_RESPONSE_BODY_BYTES) {
throw new IllegalArgumentException("响应体过大: " + body.length() + " bytes");
}
}
private static void writeLimited(StringWriter writer, char[] buffer, int len) throws IOException {
if (writer.getBuffer().length() + len > MAX_DECOMPRESSED_CHARS) {
throw new IOException("解压后响应体过大");
}
writer.write(buffer, 0, len);
}
// -------------------- 通用解压分发 -------------------- // -------------------- 通用解压分发 --------------------
private static String decompress(Buffer compressed, String encoding) throws IOException { private static String decompress(Buffer compressed, String encoding) throws IOException {
return switch (encoding.toLowerCase()) { return switch (encoding.toLowerCase()) {
case "gzip" -> decompressGzip(compressed); case "gzip" -> decompressGzip(compressed);
case "deflate" -> decompressDeflate(compressed); case "deflate" -> decompressDeflate(compressed);
case "br" -> decompressBrotli(compressed); case "br" -> decompressBrotli(compressed);
case "zstd" -> compressed.toString(StandardCharsets.UTF_8); // 暂时返回原始内容 case "zstd" -> throw new UnsupportedOperationException("不支持的 Content-Encoding: zstd");
default -> throw new UnsupportedOperationException("不支持的 Content-Encoding: " + encoding); default -> throw new UnsupportedOperationException("不支持的 Content-Encoding: " + encoding);
}; };
} }
@@ -74,7 +129,7 @@ public class HttpResponseHelper {
char[] buffer = new char[4096]; char[] buffer = new char[4096];
int n; int n;
while ((n = isr.read(buffer)) != -1) { while ((n = isr.read(buffer)) != -1) {
writer.write(buffer, 0, n); writeLimited(writer, buffer, n);
} }
return writer.toString(); return writer.toString();
} }
@@ -99,7 +154,7 @@ public class HttpResponseHelper {
char[] buffer = new char[4096]; char[] buffer = new char[4096];
int n; int n;
while ((n = isr.read(buffer)) != -1) { while ((n = isr.read(buffer)) != -1) {
writer.write(buffer, 0, n); writeLimited(writer, buffer, n);
} }
return writer.toString(); return writer.toString();
} }
@@ -115,7 +170,7 @@ public class HttpResponseHelper {
char[] buffer = new char[4096]; char[] buffer = new char[4096];
int n; int n;
while ((n = isr.read(buffer)) != -1) { while ((n = isr.read(buffer)) != -1) {
writer.write(buffer, 0, n); writeLimited(writer, buffer, n);
} }
return writer.toString(); return writer.toString();
} }
@@ -1,10 +1,12 @@
package cn.qaiu.util; package cn.qaiu.util;
import cn.qaiu.WebClientVertxInit;
import io.vertx.core.MultiMap; import io.vertx.core.MultiMap;
import io.vertx.core.Vertx;
import io.vertx.core.http.impl.headers.HeadersMultiMap; import io.vertx.core.http.impl.headers.HeadersMultiMap;
import io.vertx.ext.web.client.WebClient; import io.vertx.ext.web.client.WebClient;
import io.vertx.ext.web.client.WebClientSession; import io.vertx.ext.web.client.WebClientSession;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.HashMap; import java.util.HashMap;
import java.util.Map; import java.util.Map;
@@ -15,6 +17,12 @@ import java.util.ArrayList;
import java.util.List; import java.util.List;
public class IpExtractor { public class IpExtractor {
private static final Logger log = LoggerFactory.getLogger(IpExtractor.class);
// 使用共享的 Vertx 实例避免每次调用创建新实例导致资源泄漏
private static final WebClient SHARED_CLIENT = WebClient.create(WebClientVertxInit.get());
private static final WebClientSession SHARED_SESSION = WebClientSession.create(SHARED_CLIENT);
public static void main(String[] args) throws InterruptedException { public static void main(String[] args) throws InterruptedException {
@@ -39,12 +47,10 @@ public class IpExtractor {
headers.add("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"); headers.add("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36");
headers.add("Content-Type", "application/x-www-form-urlencoded"); headers.add("Content-Type", "application/x-www-form-urlencoded");
WebClient client = WebClient.create(Vertx.vertx()); SHARED_SESSION.getAbs("https://ip.ihuan.me").putHeaders(headers).send().onSuccess(res->{
WebClientSession webClientSession = WebClientSession.create(client); log.debug("response: {}", res.toString());
webClientSession.getAbs("https://ip.ihuan.me").putHeaders(headers).send().onSuccess(res->{ SHARED_SESSION.getAbs("https://ip.ihuan.me").putHeaders(headers).send().onSuccess(res2->{
System.out.println(res.toString()); log.debug("response2: {}", res2.toString());
webClientSession.getAbs("https://ip.ihuan.me").putHeaders(headers).send().onSuccess(res2->{
System.out.println(res2.toString());
}); });
}); });
@@ -21,11 +21,11 @@ import static cn.qaiu.util.AESUtils.encrypt;
*/ */
public class JsExecUtils { public class JsExecUtils {
private static final Invocable inv; private static final Invocable inv;
private static final ScriptEngineManager ENGINE_MANAGER = new ScriptEngineManager();
// 初始化脚本引擎 // 初始化脚本引擎
static { static {
ScriptEngineManager engineManager = new ScriptEngineManager(); ScriptEngine engine = ENGINE_MANAGER.getEngineByName("JavaScript"); // 得到脚本引擎
ScriptEngine engine = engineManager.getEngineByName("JavaScript"); // 得到脚本引擎
try { try {
engine.eval(JsContent.ye123); engine.eval(JsContent.ye123);
@@ -45,30 +45,35 @@ public class JsExecUtils {
} }
/** /**
* 调用执行蓝奏云js文件 * 调用执行蓝奏云js文件每次动态JS代码无法复用引擎
* 注意使用后清理引擎引用帮助 GC 回收 Nashorn 引擎内部资源
*/ */
public static ScriptObjectMirror executeDynamicJs(String jsText, String funName) throws ScriptException, public static ScriptObjectMirror executeDynamicJs(String jsText, String funName) throws ScriptException,
NoSuchMethodException { NoSuchMethodException {
ScriptEngineManager engineManager = new ScriptEngineManager(); ScriptEngine engine = ENGINE_MANAGER.getEngineByName("JavaScript"); // 得到脚本引擎
ScriptEngine engine = engineManager.getEngineByName("JavaScript"); // 得到脚本引擎 try {
engine.eval(JsContent.lz + "\n" + jsText); engine.eval(JsContent.lz + "\n" + jsText);
Invocable inv = (Invocable) engine; Invocable inv = (Invocable) engine;
//调用js中的函数 //调用js中的函数
if (StringUtils.isNotEmpty(funName)) { if (StringUtils.isNotEmpty(funName)) {
inv.invokeFunction(funName); inv.invokeFunction(funName);
} }
return (ScriptObjectMirror) engine.get("signObj"); return (ScriptObjectMirror) engine.get("signObj");
} finally {
// 清理引擎持有的引用帮助 GC 回收
clearEngineBindings(engine);
}
} }
/** /**
* 调用执行蓝奏云js文件 * 调用执行js文件使用缓存的 ScriptEngineManager 创建新引擎实例
* 注意使用后清理引擎引用帮助 GC 回收 Nashorn 引擎内部资源
*/ */
public static Object executeOtherJs(String jsText, String funName, Object ... args) throws ScriptException, public static Object executeOtherJs(String jsText, String funName, Object ... args) throws ScriptException,
NoSuchMethodException { NoSuchMethodException {
ScriptEngineManager engineManager = new ScriptEngineManager(); ScriptEngine engine = ENGINE_MANAGER.getEngineByName("JavaScript"); // 得到脚本引擎
ScriptEngine engine = engineManager.getEngineByName("JavaScript"); // 得到脚本引擎 try {
engine.eval(jsText); engine.eval(jsText);
Invocable inv = (Invocable) engine; Invocable inv = (Invocable) engine;
//调用js中的函数 //调用js中的函数
@@ -76,6 +81,27 @@ public class JsExecUtils {
return inv.invokeFunction(funName, args); return inv.invokeFunction(funName, args);
} }
throw new ScriptException("funName is null"); throw new ScriptException("funName is null");
} finally {
// 清理引擎持有的引用帮助 GC 回收
clearEngineBindings(engine);
}
}
/**
* 清理 ScriptEngine bindings帮助 GC 回收 Nashorn 引擎资源
*/
private static void clearEngineBindings(ScriptEngine engine) {
try {
if (engine != null) {
// 清理全局 bindings
var bindings = engine.getBindings(javax.script.ScriptContext.ENGINE_SCOPE);
if (bindings != null) {
bindings.clear();
}
}
} catch (Exception ignored) {
// 清理失败不影响主流程
}
} }
public static String getKwSign(String s, String pwd) { public static String getKwSign(String s, String pwd) {
@@ -1,22 +1,26 @@
package cn.qaiu.util; package cn.qaiu.util;
import cn.qaiu.WebClientVertxInit;
import io.vertx.core.AsyncResult; import io.vertx.core.AsyncResult;
import io.vertx.core.MultiMap; import io.vertx.core.MultiMap;
import io.vertx.core.Vertx;
import io.vertx.core.buffer.Buffer; import io.vertx.core.buffer.Buffer;
import io.vertx.core.http.impl.headers.HeadersMultiMap; import io.vertx.core.http.impl.headers.HeadersMultiMap;
import io.vertx.ext.web.client.HttpResponse; import io.vertx.ext.web.client.HttpResponse;
import io.vertx.ext.web.client.WebClient; import io.vertx.ext.web.client.WebClient;
import io.vertx.ext.web.client.WebClientSession; import io.vertx.ext.web.client.WebClientSession;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class ReqIpUtil { public class ReqIpUtil {
public static String BASE_URL = "https://ip.ihuan.me"; private static final Logger log = LoggerFactory.getLogger(ReqIpUtil.class);
public static String BASE_URL_TEMPLATE = BASE_URL + "/{path}";
public static final String BASE_URL = "https://ip.ihuan.me";
public static final String BASE_URL_TEMPLATE = BASE_URL + "/{path}";
// GET https://ip.ihuan.me/mouse.do -> $("input[name='key']").val("30b4975b5547fed806bd2b9caa18485a"); // GET https://ip.ihuan.me/mouse.do -> $("input[name='key']").val("30b4975b5547fed806bd2b9caa18485a");
public static String PATH1 = "mouse.do"; public static final String PATH1 = "mouse.do";
public static String PATH2 = "tqdl.html"; public static final String PATH2 = "tqdl.html";
// 创建请求头Map // 创建请求头Map
static MultiMap headers = new HeadersMultiMap(); static MultiMap headers = new HeadersMultiMap();
@@ -43,30 +47,28 @@ public class ReqIpUtil {
} }
// 使用共享的 Vertx 实例和 WebClient避免每次创建新实例导致资源泄漏
Vertx vertx = Vertx.vertx(); private static final WebClient WEB_CLIENT = WebClient.create(WebClientVertxInit.get());
WebClient webClient = WebClient.create(vertx); private static final WebClientSession WEB_CLIENT_SESSION = WebClientSession.create(WEB_CLIENT);
// 发送GET请求
WebClientSession webClientSession = WebClientSession.create(webClient);
public void exec() { public void exec() {
webClientSession.getAbs(BASE_URL) WEB_CLIENT_SESSION.getAbs(BASE_URL)
.putHeaders(headers) // 将请求头Map添加到请求中 .putHeaders(headers) // 将请求头Map添加到请求中
.send(this::next); .send(this::next);
} }
void next(AsyncResult<HttpResponse<Buffer>> response) { void next(AsyncResult<HttpResponse<Buffer>> response) {
if (response.failed()) { if (response.failed()) {
response.cause().printStackTrace(); log.error("请求失败", response.cause());
} else { } else {
HttpResponse<Buffer> res = response.result(); HttpResponse<Buffer> res = response.result();
System.out.println("Received response with status code " + res.statusCode()); log.debug("Received response with status code {}", res.statusCode());
System.out.println("Body: " + res.body()); log.debug("Body: {}", res.body());
webClientSession.getAbs(BASE_URL_TEMPLATE).setTemplateParam("path", PATH1) WEB_CLIENT_SESSION.getAbs(BASE_URL_TEMPLATE).setTemplateParam("path", PATH1)
.putHeaders(headers) // 将请求头Map添加到请求中 .putHeaders(headers) // 将请求头Map添加到请求中
.send(response2 -> { .send(response2 -> {
System.out.println(response2.result().bodyAsString()); log.debug("response2: {}", response2.result().bodyAsString());
}); });
} }
@@ -2,6 +2,9 @@ package cn.qaiu.util;
import org.apache.commons.lang3.StringUtils; import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.net.URL; import java.net.URL;
import java.net.URLDecoder; import java.net.URLDecoder;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
@@ -10,6 +13,8 @@ import java.util.Map;
public class URLUtil { public class URLUtil {
private static final Logger LOGGER = LoggerFactory.getLogger(URLUtil.class);
private final Map<String, String> queryParams = new HashMap<>(); private final Map<String, String> queryParams = new HashMap<>();
// 构造函数传入URL并解析参数 // 构造函数传入URL并解析参数
@@ -31,7 +36,7 @@ public class URLUtil {
} }
} }
} catch (Exception e) { } catch (Exception e) {
e.printStackTrace(); LOGGER.error("URL解析失败: {}", url, e);
} }
} }
@@ -0,0 +1,84 @@
package cn.qaiu.util;
import org.apache.commons.lang3.StringUtils;
import java.util.HashMap;
import java.util.Map;
/**
* 123分享链接子域工具将分享 key 的前半段解码为数字 uid
*/
public final class YeShareHostUtil {
private static final String CODE62 = "Tvd3hHA9QEkom14xpfaBJIMwgFYGPXn2sWCNORDr80KuUSl7bZcetizL5q6yVj";
private static final long MAX_SAFE_INTEGER = 9007199254740991L;
private static final Map<Character, Integer> DECODE_MAP = new HashMap<>();
static {
for (int i = 0; i < CODE62.length(); i++) {
DECODE_MAP.put(CODE62.charAt(i), i);
}
}
private YeShareHostUtil() {
}
public static String getNumericSubdomainIdByShareKey(String shareKey) {
String normalized = normalizeShareKey(shareKey);
if (StringUtils.isBlank(normalized)) {
return "";
}
int split = normalized.indexOf('-');
if (split <= 0) {
return "";
}
String encodedUid = normalized.substring(0, split);
Long uid = decodeBase62LittleEndian(encodedUid);
return uid == null ? "" : String.valueOf(uid);
}
public static String normalizeShareKey(String shareKey) {
if (StringUtils.isBlank(shareKey)) {
return "";
}
String key = shareKey.trim();
int queryIndex = key.indexOf('?');
if (queryIndex >= 0) {
key = key.substring(0, queryIndex);
}
int slashIndex = key.lastIndexOf('/');
if (slashIndex >= 0 && slashIndex < key.length() - 1) {
key = key.substring(slashIndex + 1);
}
if (key.endsWith(".html")) {
key = key.substring(0, key.length() - 5);
}
return key;
}
private static Long decodeBase62LittleEndian(String value) {
if (StringUtils.isBlank(value)) {
return null;
}
long result = 0L;
for (int i = 0; i < value.length(); i++) {
Integer digit = DECODE_MAP.get(value.charAt(i));
if (digit == null) {
return null;
}
double weighted = digit * Math.pow(62, i);
if (!Double.isFinite(weighted)) {
return null;
}
long next = result + (long) weighted;
if (next <= 0 || next > MAX_SAFE_INTEGER) {
return null;
}
result = next;
}
if (result <= 0) {
return null;
}
return result;
}
}
@@ -134,8 +134,8 @@ HTTP客户端对象:
http.get(url) // GET请求 http.get(url) // GET请求
http.post(url, data) // POST请求 http.post(url, data) // POST请求
http.putHeader(name, value) // 设置请求头 http.putHeader(name, value) // 设置请求头
http.sendForm(data) // 发送表单数据 http.sendForm(url, data) // 发送表单数据
http.sendJson(data) // 发送JSON数据 http.sendJson(url, data) // 发送JSON数据
``` ```
### JsHttpResponse ### JsHttpResponse
@@ -89,9 +89,9 @@ var java;
* @property {function(): JsHttpClient} clearHeaders - 清空所有请求头保留默认头 * @property {function(): JsHttpClient} clearHeaders - 清空所有请求头保留默认头
* @property {function(): Object} getHeaders - 获取所有请求头 * @property {function(): Object} getHeaders - 获取所有请求头
* @property {function(number): JsHttpClient} setTimeout - 设置请求超时时间 * @property {function(number): JsHttpClient} setTimeout - 设置请求超时时间
* @property {function(Object): JsHttpResponse} sendForm - 发送简单表单数据 * @property {function(string, Object): JsHttpResponse} sendForm - 发送简单表单数据
* @property {function(string, Object): JsHttpResponse} sendMultipartForm - 发送multipart表单数据仅支持文本字段 * @property {function(string, Object): JsHttpResponse} sendMultipartForm - 发送multipart表单数据仅支持文本字段
* @property {function(any): JsHttpResponse} sendJson - 发送JSON数据 * @property {function(string, any): JsHttpResponse} sendJson - 发送JSON数据
* @property {function(string): string} urlEncode - URL编码静态方法 * @property {function(string): string} urlEncode - URL编码静态方法
* @property {function(string): string} urlDecode - URL解码静态方法 * @property {function(string): string} urlDecode - URL解码静态方法
*/ */
+3 -3
View File
@@ -37,10 +37,10 @@
<!-- 将文件输出设置成异步输出 --> <!-- 将文件输出设置成异步输出 -->
<appender name="ASYNC-FILE" class="ch.qos.logback.classic.AsyncAppender"> <appender name="ASYNC-FILE" class="ch.qos.logback.classic.AsyncAppender">
<!-- 不丢失日志.默认的,如果队列的80%已满,则会丢弃TRACT、DEBUGINFO级别日志 --> <!-- 队列剩余 20% 时开始丢弃 TRACE/DEBUG/INFO 级别日志,避免阻塞调用线程 -->
<discardingThreshold>0</discardingThreshold> <discardingThreshold>20</discardingThreshold>
<!-- 更改默认的队列的深度,该值会影响性能.默认值为256 --> <!-- 更改默认的队列的深度,该值会影响性能.默认值为256 -->
<queueSize>256</queueSize> <queueSize>512</queueSize>
<!-- 添加附加的appender,最多只能添加一个 --> <!-- 添加附加的appender,最多只能添加一个 -->
<appender-ref ref="FILE"/> <appender-ref ref="FILE"/>
</appender> </appender>
@@ -8,6 +8,8 @@ import cn.qaiu.parser.customjs.JsParserExecutor;
import cn.qaiu.WebClientVertxInit; import cn.qaiu.WebClientVertxInit;
import io.vertx.core.Vertx; import io.vertx.core.Vertx;
import io.vertx.core.json.JsonObject; import io.vertx.core.json.JsonObject;
import org.junit.After;
import org.junit.Before;
import org.junit.Test; import org.junit.Test;
import java.util.HashMap; import java.util.HashMap;
@@ -22,15 +24,26 @@ import java.util.Map;
*/ */
public class BaiduPhotoParserTest { public class BaiduPhotoParserTest {
private Vertx vertx;
@Before
public void setUp() {
vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
}
@After
public void tearDown() {
if (vertx != null) {
vertx.close();
}
}
@Test @Test
public void testBaiduPhotoParserRegistration() { public void testBaiduPhotoParserRegistration() {
// 清理注册表 // 清理注册表
CustomParserRegistry.clear(); CustomParserRegistry.clear();
// 初始化Vertx
Vertx vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
// 检查是否加载了百度相册解析器 // 检查是否加载了百度相册解析器
CustomParserConfig config = CustomParserRegistry.get("baidu_photo"); CustomParserConfig config = CustomParserRegistry.get("baidu_photo");
assert config != null : "百度相册解析器未加载"; assert config != null : "百度相册解析器未加载";
@@ -45,10 +58,6 @@ public class BaiduPhotoParserTest {
// 清理注册表 // 清理注册表
CustomParserRegistry.clear(); CustomParserRegistry.clear();
// 初始化Vertx
Vertx vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
try { try {
// 创建解析器 - 测试文件分享链接 // 创建解析器 - 测试文件分享链接
IPanTool tool = ParserCreate.fromType("baidu_photo") IPanTool tool = ParserCreate.fromType("baidu_photo")
@@ -77,10 +86,6 @@ public class BaiduPhotoParserTest {
// 清理注册表 // 清理注册表
CustomParserRegistry.clear(); CustomParserRegistry.clear();
// 初始化Vertx
Vertx vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
try { try {
// 创建解析器 - 测试文件夹分享链接 // 创建解析器 - 测试文件夹分享链接
IPanTool tool = ParserCreate.fromType("baidu_photo") IPanTool tool = ParserCreate.fromType("baidu_photo")
@@ -109,10 +114,6 @@ public class BaiduPhotoParserTest {
// 清理注册表 // 清理注册表
CustomParserRegistry.clear(); CustomParserRegistry.clear();
// 初始化Vertx
Vertx vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
try { try {
IPanTool tool = ParserCreate.fromType("baidu_photo") IPanTool tool = ParserCreate.fromType("baidu_photo")
// 分享key PPgOEodBVE // 分享key PPgOEodBVE
@@ -167,10 +168,6 @@ public class BaiduPhotoParserTest {
// 清理注册表 // 清理注册表
CustomParserRegistry.clear(); CustomParserRegistry.clear();
// 初始化Vertx
Vertx vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
try { try {
// 创建ShareLinkInfo // 创建ShareLinkInfo
Map<String, Object> otherParam = new HashMap<>(); Map<String, Object> otherParam = new HashMap<>();
@@ -7,6 +7,8 @@ import cn.qaiu.parser.custom.CustomParserRegistry;
import cn.qaiu.parser.customjs.JsParserExecutor; import cn.qaiu.parser.customjs.JsParserExecutor;
import cn.qaiu.WebClientVertxInit; import cn.qaiu.WebClientVertxInit;
import io.vertx.core.Vertx; import io.vertx.core.Vertx;
import org.junit.After;
import org.junit.Before;
import org.junit.Test; import org.junit.Test;
import java.util.HashMap; import java.util.HashMap;
@@ -21,15 +23,26 @@ import java.util.Map;
*/ */
public class JsParserTest { public class JsParserTest {
private Vertx vertx;
@Before
public void setUp() {
vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
}
@After
public void tearDown() {
if (vertx != null) {
vertx.close();
}
}
@Test @Test
public void testJsParserRegistration() { public void testJsParserRegistration() {
// 清理注册表 // 清理注册表
CustomParserRegistry.clear(); CustomParserRegistry.clear();
// 初始化Vertx
Vertx vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
// 检查是否加载了JavaScript解析器 // 检查是否加载了JavaScript解析器
CustomParserConfig config = CustomParserRegistry.get("demo_js"); CustomParserConfig config = CustomParserRegistry.get("demo_js");
assert config != null : "JavaScript解析器未加载"; assert config != null : "JavaScript解析器未加载";
@@ -44,10 +57,6 @@ public class JsParserTest {
// 清理注册表 // 清理注册表
CustomParserRegistry.clear(); CustomParserRegistry.clear();
// 初始化Vertx
Vertx vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
try { try {
// 创建解析器 // 创建解析器
IPanTool tool = ParserCreate.fromType("demo_js") IPanTool tool = ParserCreate.fromType("demo_js")
@@ -75,10 +84,6 @@ public class JsParserTest {
// 清理注册表 // 清理注册表
CustomParserRegistry.clear(); CustomParserRegistry.clear();
// 初始化Vertx
Vertx vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
try { try {
// 创建解析器 // 创建解析器
IPanTool tool = ParserCreate.fromType("demo_js") IPanTool tool = ParserCreate.fromType("demo_js")
@@ -115,10 +120,6 @@ public class JsParserTest {
// 清理注册表 // 清理注册表
CustomParserRegistry.clear(); CustomParserRegistry.clear();
// 初始化Vertx
Vertx vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
try { try {
// 创建ShareLinkInfo // 创建ShareLinkInfo
Map<String, Object> otherParam = new HashMap<>(); Map<String, Object> otherParam = new HashMap<>();
@@ -1,6 +1,7 @@
package cn.qaiu.parser; package cn.qaiu.parser;
import cn.qaiu.entity.ShareLinkInfo; import cn.qaiu.entity.ShareLinkInfo;
import cn.qaiu.util.YeShareHostUtil;
import org.junit.Test; import org.junit.Test;
import java.util.Arrays; import java.util.Arrays;
@@ -157,6 +158,26 @@ public class PanDomainTemplateTest {
assertEquals("somekey", m5.group("KEY")); assertEquals("somekey", m5.group("KEY"));
} }
@Test
public void testYeRedirectPatternAndUidDecode() {
Pattern yePattern = PanDomainTemplate.YE.getPattern();
Matcher oldUrl = yePattern.matcher("https://www.123pan.com/s/lN7UVv-pbYJ");
assertTrue("YE should match old 123pan share URL", oldUrl.find());
assertEquals("lN7UVv-pbYJ", oldUrl.group("KEY"));
Matcher redirectedUrl = yePattern.matcher("https://1813382308.mshare.123pan.cn/123pan/lN7UVv-pbYJ");
assertTrue("YE should match redirected mshare URL", redirectedUrl.find());
assertEquals("lN7UVv-pbYJ", redirectedUrl.group("KEY"));
Matcher htmlUrl = yePattern.matcher("https://www.123278.com/s/lN7UVv-pbYJ.html?pwd=abcd");
assertTrue("YE should match html URL with query", htmlUrl.find());
assertEquals("lN7UVv-pbYJ", htmlUrl.group("KEY"));
assertEquals("1813382308", YeShareHostUtil.getNumericSubdomainIdByShareKey("lN7UVv-pbYJ"));
assertEquals("lN7UVv-pbYJ", YeShareHostUtil.normalizeShareKey("https://1813382308.mshare.123pan.cn/123pan/lN7UVv-pbYJ?pwd=abcd"));
}
@Test @Test
public void testLePatternFix() { public void testLePatternFix() {
Pattern lePattern = PanDomainTemplate.LE.getPattern(); Pattern lePattern = PanDomainTemplate.LE.getPattern();
@@ -302,6 +323,63 @@ public class PanDomainTemplateTest {
fsPattern.matcher("https://xxx.feishu.cn/docs/abc123").matches()); fsPattern.matcher("https://xxx.feishu.cn/docs/abc123").matches());
} }
@Test
public void testYsPatternMatching() {
Pattern ysPattern = PanDomainTemplate.YS.getPattern();
// 主域名
Matcher m1 = ysPattern.matcher("https://qaiu.ysepan.com/");
assertTrue("YS should match ysepan.com", m1.matches());
assertEquals("qaiu", m1.group("KEY"));
Matcher m2 = ysPattern.matcher("http://sohehe4.ys168.com");
assertTrue("YS should match ys168.com", m2.matches());
assertEquals("sohehe4", m2.group("KEY"));
// 备用域名
Matcher m3 = ysPattern.matcher("https://demo.cccpan.com/");
assertTrue("YS should match cccpan.com", m3.matches());
assertEquals("demo", m3.group("KEY"));
Matcher m4 = ysPattern.matcher("https://space.ysupan.com");
assertTrue("YS should match ysupan.com", m4.matches());
assertEquals("space", m4.group("KEY"));
Matcher m5 = ysPattern.matcher("https://user.uupan.net/");
assertTrue("YS should match uupan.net", m5.matches());
assertEquals("user", m5.group("KEY"));
Matcher m6 = ysPattern.matcher("https://ok.ysok.net");
assertTrue("YS should match ysok.net", m6.matches());
assertEquals("ok", m6.group("KEY"));
// 非空间子域 / 非白名单域名
assertFalse("YS should NOT match www.ysepan.com",
ysPattern.matcher("https://www.ysepan.com/").matches());
assertFalse("YS should NOT match api host c6.ysepan.com",
ysPattern.matcher("https://c6.ysepan.com/api/ml/mldq").matches());
assertFalse("YS should NOT match CDN ys-c.ysepan.com",
ysPattern.matcher("https://ys-c.ysepan.com/wap/qaiu/x").matches());
assertFalse("YS should NOT match unrelated domain",
ysPattern.matcher("https://qaiu.evil.com/").matches());
assertFalse("YS should NOT match ysepan.com without space subdomain",
ysPattern.matcher("https://ysepan.com/").matches());
}
@Test
public void testYsFromShareUrl() {
ParserCreate parserCreate = ParserCreate.fromShareUrl("https://qaiu.ysepan.com/");
ShareLinkInfo info = parserCreate.getShareLinkInfo();
assertNotNull(info);
assertEquals("ys", info.getType());
assertEquals("永硕E盘", info.getPanName());
assertEquals("qaiu", info.getShareKey());
ParserCreate backup = ParserCreate.fromShareUrl("https://demo.cccpan.com/");
assertEquals("ys", backup.getShareLinkInfo().getType());
assertEquals("demo", backup.getShareLinkInfo().getShareKey());
}
@Test @Test
public void testFsFromShareUrl() { public void testFsFromShareUrl() {
// 测试文件链接解析 // 测试文件链接解析
@@ -7,6 +7,8 @@ import cn.qaiu.parser.ParserCreate;
import cn.qaiu.parser.custom.CustomParserConfig; import cn.qaiu.parser.custom.CustomParserConfig;
import cn.qaiu.parser.custom.CustomParserRegistry; import cn.qaiu.parser.custom.CustomParserRegistry;
import io.vertx.core.Vertx; import io.vertx.core.Vertx;
import org.junit.After;
import org.junit.Before;
import org.junit.Test; import org.junit.Test;
import org.slf4j.Logger; import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
@@ -19,12 +21,23 @@ public class JsFetchBridgeTest {
private static final Logger log = LoggerFactory.getLogger(JsFetchBridgeTest.class); private static final Logger log = LoggerFactory.getLogger(JsFetchBridgeTest.class);
private Vertx vertx;
@Before
public void setUp() {
vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
}
@After
public void tearDown() {
if (vertx != null) {
vertx.close();
}
}
@Test @Test
public void testFetchPolyfillLoaded() { public void testFetchPolyfillLoaded() {
// 初始化Vertx
Vertx vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
// 清理注册表 // 清理注册表
CustomParserRegistry.clear(); CustomParserRegistry.clear();
@@ -83,10 +96,6 @@ public class JsFetchBridgeTest {
@Test @Test
public void testPromiseBasicUsage() { public void testPromiseBasicUsage() {
// 初始化Vertx
Vertx vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
// 清理注册表 // 清理注册表
CustomParserRegistry.clear(); CustomParserRegistry.clear();
@@ -0,0 +1,312 @@
package cn.qaiu.parser.impl;
import cn.qaiu.WebClientVertxInit;
import cn.qaiu.entity.FileInfo;
import cn.qaiu.parser.PanDomainTemplate;
import cn.qaiu.parser.ParserCreate;
import cn.qaiu.util.CommonUtils;
import io.vertx.core.Vertx;
import io.vertx.core.buffer.Buffer;
import io.vertx.core.json.JsonObject;
import io.vertx.ext.web.client.WebClient;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import static org.junit.Assert.*;
/**
* 永硕E盘解析测试含示例空间联调 + 真实下载校验
*/
public class YsToolTest {
private static Vertx vertx;
private static WebClient webClient;
@BeforeClass
public static void setUpClass() {
vertx = Vertx.vertx();
WebClientVertxInit.init(vertx);
webClient = WebClient.create(vertx);
}
@AfterClass
public static void tearDownClass() {
if (webClient != null) {
webClient.close();
}
if (vertx != null) {
vertx.close();
}
}
@Test
public void testPatternMatching() {
Pattern pattern = PanDomainTemplate.YS.getPattern();
Matcher m1 = pattern.matcher("https://qaiu.ysepan.com/");
assertTrue(m1.matches());
assertEquals("qaiu", m1.group("KEY"));
Matcher m2 = pattern.matcher("http://sohehe4.ysepan.com");
assertTrue(m2.matches());
assertEquals("sohehe4", m2.group("KEY"));
Matcher m3 = pattern.matcher("https://demo.ys168.com/");
assertTrue(m3.matches());
assertEquals("demo", m3.group("KEY"));
assertTrue(pattern.matcher("https://a.cccpan.com/").matches());
assertTrue(pattern.matcher("https://a.ysupan.com").matches());
assertTrue(pattern.matcher("https://a.uupan.net/").matches());
assertTrue(pattern.matcher("https://a.ysok.net").matches());
assertFalse(pattern.matcher("https://www.ysepan.com/").matches());
assertFalse(pattern.matcher("https://c6.ysepan.com/api/ml/mldq").matches());
assertFalse(pattern.matcher("https://ys-c.ysepan.com/wap/qaiu/x/y/z").matches());
assertFalse(pattern.matcher("https://zy.ysepan.com/assets/index.js").matches());
assertFalse(pattern.matcher("https://qaiu.evil.com/").matches());
}
@Test
public void testIdentifyShareUrl() {
ParserCreate create = ParserCreate.fromShareUrl("https://qaiu.ysepan.com/");
assertEquals("ys", create.getShareLinkInfo().getType());
assertEquals("永硕E盘", create.getShareLinkInfo().getPanName());
assertEquals("qaiu", create.getShareLinkInfo().getShareKey());
}
@Test
public void testBuildDownloadUrl() {
String url = YsTool.buildDownloadUrl(
"yssl",
"A95UIe495EkSKE",
"Bc8hF8Nsbl2I4Ec6vAm5EGe9HU36iC",
"L",
"lu20.jpg");
// 与官方一致xzpz 前不加 "_"
assertEquals(
"https://ys-l.ysepan.com/wap/yssl/A95UIe495EkSKE/Bc8hF8Nsbl2I4Ec6vAm5EGe9HU36iC/lu20.jpg",
url);
assertFalse("force-download 前缀会导致 404", url.contains("/_"));
}
@Test
public void testOfficialSampleUrlRealDownload() throws Exception {
String official = "https://ys-l.ysepan.com/wap/yssl/A95UIe495EkSKE/Bc8hF8Nsbl2I4Ec6vAm5EGe9HU36iC/lu20.jpg";
String withForcePrefix = "https://ys-l.ysepan.com/wap/yssl/_A95UIe495EkSKE/Bc8hF8Nsbl2I4Ec6vAm5EGe9HU36iC/lu20.jpg";
Buffer ok = download(official, "https://yssl.ysepan.com/");
assertTrue("官方直链应能下载到 JPEG", ok.length() > 1000);
assertEquals((byte) 0xFF, ok.getByte(0));
assertEquals((byte) 0xD8, ok.getByte(1));
int forceStatus = webClient.getAbs(withForcePrefix)
.putHeader("User-Agent", "Mozilla/5.0")
.putHeader("Referer", "https://yssl.ysepan.com/")
.send()
.toCompletionStage().toCompletableFuture()
.get(30, TimeUnit.SECONDS)
.statusCode();
assertNotEquals("带 _ 前缀的直链应失败(文件不存在)", 200, forceStatus);
}
@Test
public void testQaiuSpaceFileListAndRealDownload() throws Exception {
ParserCreate create = ParserCreate.fromShareUrl("https://qaiu.ysepan.com/");
create.getShareLinkInfo().setSharePassword("qaiuys168");
create.getShareLinkInfo().getOtherParam().put("domainName", "http://localhost");
List<FileInfo> dirs = create.createTool().parseFileList()
.toCompletionStage().toCompletableFuture()
.get(60, TimeUnit.SECONDS);
assertNotNull(dirs);
assertFalse("目录列表不应为空", dirs.isEmpty());
assertEquals("folder", dirs.get(0).getFileType());
String dirId = dirs.get(0).getFileId();
assertNotNull(dirId);
ParserCreate filesCreate = ParserCreate.fromShareUrl("https://qaiu.ysepan.com/");
filesCreate.getShareLinkInfo().setSharePassword("qaiuys168");
filesCreate.getShareLinkInfo().getOtherParam().put("domainName", "http://localhost");
filesCreate.getShareLinkInfo().getOtherParam().put("dirId", dirId);
List<FileInfo> files = filesCreate.createTool().parseFileList()
.toCompletionStage().toCompletableFuture()
.get(60, TimeUnit.SECONDS);
assertNotNull(files);
FileInfo zip = files.stream()
.filter(f -> "file".equals(f.getFileType()))
.filter(f -> f.getFileName() != null && f.getFileName().contains("Pycharm"))
.findFirst()
.orElse(null);
assertNotNull("应能列出 Pycharm 压缩包", zip);
assertTrue(zip.getSize() > 0);
assertNotNull(zip.getParserUrl());
String param = zip.getParserUrl().substring(zip.getParserUrl().lastIndexOf('/') + 1);
JsonObject paramJson = new JsonObject(CommonUtils.urlBase64Decode(param));
assertFalse("downloadUrl 不应含 force 前缀",
paramJson.getString("downloadUrl", "").contains("/_"));
ParserCreate byId = ParserCreate.fromType("ys").shareKey("qaiu");
byId.getShareLinkInfo().setSharePassword("qaiuys168");
byId.getShareLinkInfo().getOtherParam().put("paramJson", paramJson);
String downloadUrl = byId.createTool().parseById()
.toCompletionStage().toCompletableFuture()
.get(60, TimeUnit.SECONDS);
assertNotNull(downloadUrl);
assertFalse("解析直链不应含 _xzpz 前缀", downloadUrl.matches(".*/_[^/]+/.*"));
assertTrue(downloadUrl.contains("ysepan.com"));
Buffer body = download(downloadUrl, "https://qaiu.ysepan.com/");
assertEquals("真实下载大小应与列表一致", zip.getSize().longValue(), body.length());
// ZIP magic: PK
assertEquals('P', (char) body.getByte(0));
assertEquals('K', (char) body.getByte(1));
System.out.println("qaiu real download ok, url=" + downloadUrl + ", size=" + body.length());
}
@Test
public void testFufu1ZmlHierarchyAndUrlItems() throws Exception {
// https://fufu1.ysepan.com/ 无密码游戏3 下应按 zml 展示子目录再进子目录才是夸克/百度链接
ParserCreate create = ParserCreate.fromShareUrl("https://fufu1.ysepan.com/");
create.getShareLinkInfo().getOtherParam().put("domainName", "http://localhost");
List<FileInfo> roots = create.createTool().parseFileList()
.toCompletionStage().toCompletableFuture()
.get(60, TimeUnit.SECONDS);
assertNotNull(roots);
FileInfo game3 = roots.stream()
.filter(f -> "folder".equals(f.getFileType()))
.filter(f -> "游戏3".equals(f.getFileName()))
.findFirst()
.orElse(null);
assertNotNull("应有目录 游戏3", game3);
ParserCreate level2 = ParserCreate.fromShareUrl("https://fufu1.ysepan.com/");
level2.getShareLinkInfo().getOtherParam().put("domainName", "http://localhost");
level2.getShareLinkInfo().getOtherParam().put("dirId", game3.getFileId());
List<FileInfo> subdirs = level2.createTool().parseFileList()
.toCompletionStage().toCompletableFuture()
.get(60, TimeUnit.SECONDS);
assertNotNull(subdirs);
assertTrue("游戏3 下应是子目录列表", subdirs.size() > 10);
assertTrue("游戏3 下列表应全是 folder(zml 子目录)",
subdirs.stream().allMatch(f -> "folder".equals(f.getFileType())));
FileInfo sample = subdirs.stream()
.filter(f -> f.getFileName() != null && f.getFileName().contains("我是未来"))
.findFirst()
.orElse(subdirs.get(0));
// parserUrl 提取 zml或直接用 fileName
ParserCreate level3 = ParserCreate.fromShareUrl("https://fufu1.ysepan.com/");
level3.getShareLinkInfo().getOtherParam().put("domainName", "http://localhost");
level3.getShareLinkInfo().getOtherParam().put("dirId", game3.getFileId());
level3.getShareLinkInfo().getOtherParam().put("zml", sample.getFileName());
List<FileInfo> links = level3.createTool().parseFileList()
.toCompletionStage().toCompletableFuture()
.get(60, TimeUnit.SECONDS);
assertNotNull(links);
assertFalse(links.isEmpty());
assertTrue("子目录内应有 url 类型",
links.stream().anyMatch(f -> "url".equals(f.getFileType())));
assertTrue("应包含夸克/百度链接名",
links.stream().anyMatch(f -> "夸克".equals(f.getFileName()) || "百度".equals(f.getFileName())));
assertFalse("空占位 URL 不应出现",
links.stream().anyMatch(f -> f.getFileName() == null || f.getFileName().isBlank()));
assertTrue("URL 条目应带 previewUrl",
links.stream().filter(f -> "url".equals(f.getFileType()))
.allMatch(f -> f.getPreviewUrl() != null && f.getPreviewUrl().startsWith("http")));
System.out.println("fufu1 hierarchy ok: 游戏3 -> " + sample.getFileName()
+ " -> " + links.stream().map(FileInfo::getFileName).toList());
}
@Test
public void testSohehe4SpaceDirectories() throws Exception {
ParserCreate create = ParserCreate.fromShareUrl("https://sohehe4.ysepan.com/");
create.getShareLinkInfo().setSharePassword("1234");
create.getShareLinkInfo().getOtherParam().put("domainName", "http://localhost");
List<FileInfo> dirs = create.createTool().parseFileList()
.toCompletionStage().toCompletableFuture()
.get(60, TimeUnit.SECONDS);
assertNotNull(dirs);
assertTrue("sohehe4 应有多个目录", dirs.size() >= 3);
assertTrue(dirs.stream().allMatch(d -> "folder".equals(d.getFileType())));
String dirId = dirs.stream()
.filter(d -> d.getFileName() != null && d.getFileName().contains("留言"))
.map(FileInfo::getFileId)
.findFirst()
.orElse(dirs.get(0).getFileId());
ParserCreate filesCreate = ParserCreate.fromShareUrl("https://sohehe4.ysepan.com/");
filesCreate.getShareLinkInfo().setSharePassword("1234");
filesCreate.getShareLinkInfo().getOtherParam().put("domainName", "http://localhost");
filesCreate.getShareLinkInfo().getOtherParam().put("dirId", dirId);
List<FileInfo> files = filesCreate.createTool().parseFileList()
.toCompletionStage().toCompletableFuture()
.get(60, TimeUnit.SECONDS);
assertNotNull(files);
assertFalse(files.isEmpty());
FileInfo file = files.stream()
.filter(f -> "file".equals(f.getFileType()))
.filter(f -> f.getSize() != null && f.getSize() > 0 && f.getSize() < 5_000_000)
.findFirst()
.orElse(null);
if (file != null) {
String param = file.getParserUrl().substring(file.getParserUrl().lastIndexOf('/') + 1);
JsonObject paramJson = new JsonObject(CommonUtils.urlBase64Decode(param));
String downloadUrl = paramJson.getString("downloadUrl");
assertNotNull(downloadUrl);
assertFalse(downloadUrl.contains("/_"));
Buffer body = download(downloadUrl, "https://sohehe4.ysepan.com/");
assertEquals(file.getSize().longValue(), body.length());
System.out.println("sohehe4 real download ok, file=" + file.getFileName()
+ ", size=" + body.length());
} else {
System.out.println("sohehe4 dir=" + dirId + " entries=" + files.size()
+ " (no small file for real download sample)");
}
}
private static Buffer download(String url, String referer) throws Exception {
return webClient.getAbs(url)
.putHeader("User-Agent",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
+ "(KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36")
.putHeader("Referer", referer)
.send()
.toCompletionStage().toCompletableFuture()
.thenApply(res -> {
assertEquals("下载 HTTP 状态码应为 200: " + url, 200, res.statusCode());
Buffer body = res.body();
assertNotNull(body);
assertTrue("下载内容为空: " + url, body.length() > 0);
String ct = res.getHeader("Content-Type");
assertFalse("不应返回 HTML 错误页: " + url,
ct != null && ct.toLowerCase().contains("text/html"));
return body;
})
.get(60, TimeUnit.SECONDS);
}
}
@@ -11,7 +11,7 @@
## 测试配置 ## 测试配置
### 小飞机网盘 ✅ ### 小飞机网盘 ✅
- **用户名**: 15764091073 - **用户名**: 15x
- **URL**: https://share.feijipan.com/s/ZWYoZ31c - **URL**: https://share.feijipan.com/s/ZWYoZ31c
- **文件**: 资源.rar (1.13 GB) - **文件**: 资源.rar (1.13 GB)
- **认证方式**: username/password - **认证方式**: username/password
@@ -32,7 +32,7 @@
``` ```
=== 测试小飞机网盘解析(带认证)=== === 测试小飞机网盘解析(带认证)===
分享链接: https://share.feijipan.com/s/ZWYoZ31c 分享链接: https://share.feijipan.com/s/ZWYoZ31c
用户名: 15764091073 用户名: 15x
密码: ****** 密码: ******
开始解析... 开始解析...
+329
View File
@@ -0,0 +1,329 @@
// ==FetchRuntime==
// @name Fetch API Polyfill for ES5
// @description Fetch API and Promise implementation for ES5 JavaScript engines
// @version 1.0.0
// @author QAIU
// ==============
/**
* Simple Promise implementation compatible with ES5
* Supports basic Promise functionality needed for fetch API
*/
function SimplePromise(executor) {
var state = 'pending';
var value;
var handlers = [];
var self = this;
function resolve(result) {
if (state !== 'pending') return;
state = 'fulfilled';
value = result;
handlers.forEach(handle);
handlers = [];
}
function reject(err) {
if (state !== 'pending') return;
state = 'rejected';
value = err;
handlers.forEach(handle);
handlers = [];
}
function handle(handler) {
if (state === 'pending') {
handlers.push(handler);
} else {
setTimeout(function() {
if (state === 'fulfilled' && typeof handler.onFulfilled === 'function') {
try {
var result = handler.onFulfilled(value);
if (result && typeof result.then === 'function') {
result.then(handler.resolve, handler.reject);
} else {
handler.resolve(result);
}
} catch (e) {
handler.reject(e);
}
}
if (state === 'rejected' && typeof handler.onRejected === 'function') {
try {
var result = handler.onRejected(value);
if (result && typeof result.then === 'function') {
result.then(handler.resolve, handler.reject);
} else {
handler.resolve(result);
}
} catch (e) {
handler.reject(e);
}
} else if (state === 'rejected' && !handler.onRejected) {
handler.reject(value);
}
}, 0);
}
}
this.then = function(onFulfilled, onRejected) {
return new SimplePromise(function(resolveNext, rejectNext) {
handle({
onFulfilled: onFulfilled,
onRejected: onRejected,
resolve: resolveNext,
reject: rejectNext
});
});
};
this['catch'] = function(onRejected) {
return this.then(null, onRejected);
};
this['finally'] = function(onFinally) {
return this.then(
function(value) {
return SimplePromise.resolve(onFinally()).then(function() {
return value;
});
},
function(reason) {
return SimplePromise.resolve(onFinally()).then(function() {
throw reason;
});
}
);
};
try {
executor(resolve, reject);
} catch (e) {
reject(e);
}
}
// Static methods
SimplePromise.resolve = function(value) {
if (value && typeof value.then === 'function') {
return value;
}
return new SimplePromise(function(resolve) {
resolve(value);
});
};
SimplePromise.reject = function(reason) {
return new SimplePromise(function(resolve, reject) {
reject(reason);
});
};
SimplePromise.all = function(promises) {
return new SimplePromise(function(resolve, reject) {
var results = [];
var remaining = promises.length;
if (remaining === 0) {
resolve(results);
return;
}
function handleResult(index, value) {
results[index] = value;
remaining--;
if (remaining === 0) {
resolve(results);
}
}
for (var i = 0; i < promises.length; i++) {
(function(index) {
var promise = promises[index];
if (promise && typeof promise.then === 'function') {
promise.then(
function(value) { handleResult(index, value); },
reject
);
} else {
handleResult(index, promise);
}
})(i);
}
});
};
SimplePromise.race = function(promises) {
return new SimplePromise(function(resolve, reject) {
if (promises.length === 0) {
// Per spec, Promise.race with empty array stays pending forever
return;
}
for (var i = 0; i < promises.length; i++) {
var promise = promises[i];
if (promise && typeof promise.then === 'function') {
promise.then(resolve, reject);
} else {
resolve(promise);
return;
}
}
});
};
// Make Promise global if not already defined
if (typeof Promise === 'undefined') {
var Promise = SimplePromise;
}
/**
* Response object that mimics the Fetch API Response
*/
function FetchResponse(jsHttpResponse) {
this._jsResponse = jsHttpResponse;
this.status = jsHttpResponse.statusCode();
this.ok = this.status >= 200 && this.status < 300;
// Map HTTP status codes to standard status text
var statusTexts = {
200: 'OK',
201: 'Created',
204: 'No Content',
301: 'Moved Permanently',
302: 'Found',
304: 'Not Modified',
400: 'Bad Request',
401: 'Unauthorized',
403: 'Forbidden',
404: 'Not Found',
405: 'Method Not Allowed',
408: 'Request Timeout',
409: 'Conflict',
410: 'Gone',
500: 'Internal Server Error',
501: 'Not Implemented',
502: 'Bad Gateway',
503: 'Service Unavailable',
504: 'Gateway Timeout'
};
this.statusText = statusTexts[this.status] || (this.ok ? 'OK' : 'Error');
this.headers = {
get: function(name) {
return jsHttpResponse.header(name);
},
has: function(name) {
return jsHttpResponse.header(name) !== null;
},
entries: function() {
var headerMap = jsHttpResponse.headers();
var entries = [];
for (var key in headerMap) {
if (headerMap.hasOwnProperty(key)) {
entries.push([key, headerMap[key]]);
}
}
return entries;
}
};
}
FetchResponse.prototype.text = function() {
var body = this._jsResponse.body();
return SimplePromise.resolve(body || '');
};
FetchResponse.prototype.json = function() {
var self = this;
return this.text().then(function(text) {
try {
return JSON.parse(text);
} catch (e) {
throw new Error('Invalid JSON: ' + e.message);
}
});
};
FetchResponse.prototype.arrayBuffer = function() {
var bytes = this._jsResponse.bodyBytes();
return SimplePromise.resolve(bytes);
};
FetchResponse.prototype.blob = function() {
// Blob not supported in ES5, return bytes
return this.arrayBuffer();
};
/**
* Fetch API implementation using JavaFetch bridge
* @param {string} url - Request URL
* @param {Object} options - Fetch options (method, headers, body, etc.)
* @returns {Promise<FetchResponse>}
*/
function fetch(url, options) {
return new SimplePromise(function(resolve, reject) {
try {
// Parse options
options = options || {};
var method = (options.method || 'GET').toUpperCase();
var headers = options.headers || {};
var body = options.body;
// Prepare request options for JavaFetch
var requestOptions = {
method: method,
headers: {}
};
// Convert headers to simple object
if (headers) {
if (typeof headers.forEach === 'function') {
// Headers object
headers.forEach(function(value, key) {
requestOptions.headers[key] = value;
});
} else if (typeof headers === 'object') {
// Plain object
for (var key in headers) {
if (headers.hasOwnProperty(key)) {
requestOptions.headers[key] = headers[key];
}
}
}
}
// Add body if present
if (body !== undefined && body !== null) {
if (typeof body === 'string') {
requestOptions.body = body;
} else if (typeof body === 'object') {
// Assume JSON
requestOptions.body = JSON.stringify(body);
if (!requestOptions.headers['Content-Type'] && !requestOptions.headers['content-type']) {
requestOptions.headers['Content-Type'] = 'application/json';
}
}
}
// Call JavaFetch bridge
var jsHttpResponse = JavaFetch.fetch(url, requestOptions);
// Create Response object
var response = new FetchResponse(jsHttpResponse);
resolve(response);
} catch (e) {
reject(e);
}
});
}
// Export for global use
if (typeof window !== 'undefined') {
window.fetch = fetch;
window.Promise = Promise;
} else if (typeof global !== 'undefined') {
global.fetch = fetch;
global.Promise = Promise;
}
+7 -6
View File
@@ -17,7 +17,7 @@
</modules> </modules>
<properties> <properties>
<revision>0.2.1</revision> <revision>0.4.2</revision>
<java.version>17</java.version> <java.version>17</java.version>
<maven.compiler.source>17</maven.compiler.source> <maven.compiler.source>17</maven.compiler.source>
<maven.compiler.target>17</maven.compiler.target> <maven.compiler.target>17</maven.compiler.target>
@@ -25,16 +25,17 @@
<packageDirectory>${project.basedir}/web-service/target/package</packageDirectory> <packageDirectory>${project.basedir}/web-service/target/package</packageDirectory>
<!-- Vert.x 4.5.24 已包含安全修复,无需单独指定 Netty 版本 --> <!-- Vert.x 4.5.27 包含安全修复,无需单独指定 Netty 版本 -->
<vertx.version>4.5.24</vertx.version> <vertx.version>4.5.27</vertx.version>
<org.reflections.version>0.10.2</org.reflections.version> <org.reflections.version>0.10.2</org.reflections.version>
<lombok.version>1.18.38</lombok.version> <lombok.version>1.18.38</lombok.version>
<slf4j.version>2.0.16</slf4j.version> <slf4j.version>2.0.16</slf4j.version>
<commons-lang3.version>3.18.0</commons-lang3.version> <commons-lang3.version>3.18.0</commons-lang3.version>
<commons-beanutils2.version>2.0.0</commons-beanutils2.version> <commons-beanutils2.version>2.0.0</commons-beanutils2.version>
<jackson.version>2.18.6</jackson.version> <parserVersion>10.2.5</parserVersion>
<jackson.version>2.18.9</jackson.version>
<!-- Logback 最新稳定版 --> <!-- Logback 最新稳定版 -->
<logback.version>1.5.18</logback.version> <logback.version>1.5.34</logback.version>
<junit.version>4.13.2</junit.version> <junit.version>4.13.2</junit.version>
</properties> </properties>
@@ -74,7 +75,7 @@
<dependency> <dependency>
<groupId>cn.qaiu</groupId> <groupId>cn.qaiu</groupId>
<artifactId>parser</artifactId> <artifactId>parser</artifactId>
<version>10.2.5</version> <version>${parserVersion}</version>
</dependency> </dependency>
</dependencies> </dependencies>
</dependencyManagement> </dependencyManagement>
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

+4 -4
View File
@@ -1,19 +1,19 @@
{ {
"name": "nfd-web", "name": "nfd-web",
"version": "0.2.1", "version": "0.4.2",
"private": true, "private": true,
"scripts": { "scripts": {
"serve": "vue-cli-service serve", "serve": "vue-cli-service serve",
"dev": "vue-cli-service serve", "dev": "vue-cli-service serve",
"build": "vue-cli-service build && node scripts/compress-vs.js", "build": "node scripts/sync-version.js && vue-cli-service build && node scripts/compress-vs.js",
"build:no-compress": "vue-cli-service build", "build:no-compress": "node scripts/sync-version.js && vue-cli-service build",
"lint": "vue-cli-service lint" "lint": "vue-cli-service lint"
}, },
"dependencies": { "dependencies": {
"@element-plus/icons-vue": "^2.3.1", "@element-plus/icons-vue": "^2.3.1",
"@monaco-editor/loader": "^1.4.0", "@monaco-editor/loader": "^1.4.0",
"@vueuse/core": "^11.2.0", "@vueuse/core": "^11.2.0",
"axios": "1.13.5", "axios": "1.18.0",
"clipboard": "^2.0.11", "clipboard": "^2.0.11",
"core-js": "^3.8.3", "core-js": "^3.8.3",
"crypto-js": "^4.2.0", "crypto-js": "^4.2.0",
Binary file not shown.

Before

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 302 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 482 KiB

Some files were not shown because too many files have changed in this diff Show More