You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
494 lines
23 KiB
494 lines
23 KiB
|
1 month ago
|
# 文件存储配置
|
||
|
|
|
||
|
|
<cite>
|
||
|
|
**本文引用的文件**
|
||
|
|
- [application.yml](file://crm-app/src/main/resources/application.yml)
|
||
|
|
- [CrmAppApplication.java](file://crm-app/src/main/java/com/crm/app/CrmAppApplication.java)
|
||
|
|
- [FileProperties.java](file://crm-file/src/main/java/com/crm/file/config/FileProperties.java)
|
||
|
|
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
|
||
|
|
- [SchedulingConfig.java](file://crm-file/src/main/java/com/crm/file/config/SchedulingConfig.java)
|
||
|
|
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
|
||
|
|
- [FileApi.java](file://crm-file/src/main/java/com/crm/file/api/FileApi.java)
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
- [FileInfoServiceImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileInfoServiceImpl.java)
|
||
|
|
- [IFileInfoService.java](file://crm-file/src/main/java/com/crm/file/service/IFileInfoService.java)
|
||
|
|
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)
|
||
|
|
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)
|
||
|
|
- [FileInfo.java](file://crm-file/src/main/java/com/crm/file/domain/entity/FileInfo.java)
|
||
|
|
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
|
||
|
|
- [MultipartInitDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/MultipartInitDTO.java)
|
||
|
|
- [UploadSession.java](file://crm-file/src/main/java/com/crm/file/domain/dto/UploadSession.java)
|
||
|
|
- [FileDownloadDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/FileDownloadDTO.java)
|
||
|
|
- [FileInfoDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/FileInfoDTO.java)
|
||
|
|
- [FileConstants.java](file://crm-file/src/main/java/com/crm/file/constant/FileConstants.java)
|
||
|
|
</cite>
|
||
|
|
|
||
|
|
## 目录
|
||
|
|
1. [简介](#简介)
|
||
|
|
2. [项目结构](#项目结构)
|
||
|
|
3. [核心组件](#核心组件)
|
||
|
|
4. [架构总览](#架构总览)
|
||
|
|
5. [详细组件分析](#详细组件分析)
|
||
|
|
6. [依赖关系分析](#依赖关系分析)
|
||
|
|
7. [性能考虑](#性能考虑)
|
||
|
|
8. [故障排查指南](#故障排查指南)
|
||
|
|
9. [结论](#结论)
|
||
|
|
10. [附录](#附录)
|
||
|
|
|
||
|
|
## 简介
|
||
|
|
本文件围绕文件存储子系统,系统性梳理 MinIO 连接与存储管理、文件上传下载、分片上传流程、定时任务(断点续传清理)等能力;同时给出不同存储后端切换方法、容量监控与备份恢复建议、以及大文件处理策略与性能调优要点。文档面向开发与运维人员,兼顾可操作性与可维护性。
|
||
|
|
|
||
|
|
## 项目结构
|
||
|
|
文件存储相关代码集中在 crm-file 模块,包含配置类、控制器、服务实现、数据模型、定时任务与客户端封装等。应用入口在 crm-app 模块,配置文件位于 resources 下。
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
graph TB
|
||
|
|
subgraph "应用启动"
|
||
|
|
APP["CrmAppApplication"]
|
||
|
|
CFG["application.yml"]
|
||
|
|
end
|
||
|
|
subgraph "文件模块(crm-file)"
|
||
|
|
CTRL["FileController"]
|
||
|
|
API_IF["FileApi"]
|
||
|
|
API_IMPL["FileApiImpl"]
|
||
|
|
FILE_SVC["FileInfoServiceImpl"]
|
||
|
|
KK_CLIENT["KkFileViewClient"]
|
||
|
|
MINIO_CFG["MinioConfig"]
|
||
|
|
FILE_PROP["FileProperties"]
|
||
|
|
SCHED_CFG["SchedulingConfig"]
|
||
|
|
TASK["OrphanChunkCleanupTask"]
|
||
|
|
MODEL["FileInfo 实体"]
|
||
|
|
MAPPER["FileInfoMapper"]
|
||
|
|
DTOs["DTO: MultipartInitDTO, UploadSession, FileDownloadDTO, FileInfoDTO"]
|
||
|
|
CONST["FileConstants"]
|
||
|
|
end
|
||
|
|
APP --> CFG
|
||
|
|
CTRL --> API_IF
|
||
|
|
API_IF --> API_IMPL
|
||
|
|
API_IMPL --> FILE_SVC
|
||
|
|
FILE_SVC --> MAPPER
|
||
|
|
API_IMPL --> MINIO_CFG
|
||
|
|
API_IMPL --> FILE_PROP
|
||
|
|
API_IMPL --> KK_CLIENT
|
||
|
|
SCHED_CFG --> TASK
|
||
|
|
CTRL --> DTOs
|
||
|
|
FILE_SVC --> MODEL
|
||
|
|
API_IMPL --> CONST
|
||
|
|
```
|
||
|
|
|
||
|
|
图表来源
|
||
|
|
- [CrmAppApplication.java](file://crm-app/src/main/java/com/crm/app/CrmAppApplication.java)
|
||
|
|
- [application.yml](file://crm-app/src/main/resources/application.yml)
|
||
|
|
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
|
||
|
|
- [FileApi.java](file://crm-file/src/main/java/com/crm/file/api/FileApi.java)
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
- [FileInfoServiceImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileInfoServiceImpl.java)
|
||
|
|
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
|
||
|
|
- [FileProperties.java](file://crm-file/src/main/java/com/crm/file/config/FileProperties.java)
|
||
|
|
- [SchedulingConfig.java](file://crm-file/src/main/java/com/crm/file/config/SchedulingConfig.java)
|
||
|
|
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)
|
||
|
|
- [FileInfo.java](file://crm-file/src/main/java/com/crm/file/domain/entity/FileInfo.java)
|
||
|
|
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
|
||
|
|
- [MultipartInitDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/MultipartInitDTO.java)
|
||
|
|
- [UploadSession.java](file://crm-file/src/main/java/com/crm/file/domain/dto/UploadSession.java)
|
||
|
|
- [FileDownloadDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/FileDownloadDTO.java)
|
||
|
|
- [FileInfoDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/FileInfoDTO.java)
|
||
|
|
- [FileConstants.java](file://crm-file/src/main/java/com/crm/file/constant/FileConstants.java)
|
||
|
|
|
||
|
|
章节来源
|
||
|
|
- [CrmAppApplication.java](file://crm-app/src/main/java/com/crm/app/CrmAppApplication.java)
|
||
|
|
- [application.yml](file://crm-app/src/main/resources/application.yml)
|
||
|
|
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
|
||
|
|
- [FileApi.java](file://crm-file/src/main/java/com/crm/file/api/FileApi.java)
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
- [FileInfoServiceImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileInfoServiceImpl.java)
|
||
|
|
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
|
||
|
|
- [FileProperties.java](file://crm-file/src/main/java/com/crm/file/config/FileProperties.java)
|
||
|
|
- [SchedulingConfig.java](file://crm-file/src/main/java/com/crm/file/config/SchedulingConfig.java)
|
||
|
|
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)
|
||
|
|
- [FileInfo.java](file://crm-file/src/main/java/com/crm/file/domain/entity/FileInfo.java)
|
||
|
|
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
|
||
|
|
- [MultipartInitDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/MultipartInitDTO.java)
|
||
|
|
- [UploadSession.java](file://crm-file/src/main/java/com/crm/file/domain/dto/UploadSession.java)
|
||
|
|
- [FileDownloadDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/FileDownloadDTO.java)
|
||
|
|
- [FileInfoDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/FileInfoDTO.java)
|
||
|
|
- [FileConstants.java](file://crm-file/src/main/java/com/crm/file/constant/FileConstants.java)
|
||
|
|
|
||
|
|
## 核心组件
|
||
|
|
- 配置层
|
||
|
|
- MinioConfig:封装 MinIO 客户端初始化与连接参数(端点、访问密钥、安全协议、超时等)。
|
||
|
|
- FileProperties:集中读取文件相关配置项(如桶名、路径前缀、分片大小、超时等)。
|
||
|
|
- SchedulingConfig:启用并配置定时任务调度器。
|
||
|
|
- 接口与控制器
|
||
|
|
- FileApi:定义文件上传、下载、分片初始化、分片上传、合并、删除等接口契约。
|
||
|
|
- FileController:对外暴露 HTTP 接口,接收请求并委托给 FileApi。
|
||
|
|
- 服务层
|
||
|
|
- FileApiImpl:实现文件上传下载、分片上传合并、权限校验、元数据持久化等核心逻辑。
|
||
|
|
- FileInfoServiceImpl:文件信息 CRUD 与查询聚合。
|
||
|
|
- KkFileViewClient:在线预览客户端封装。
|
||
|
|
- 数据层
|
||
|
|
- FileInfo 实体与 FileInfoMapper:文件元数据持久化。
|
||
|
|
- 定时任务
|
||
|
|
- OrphanChunkCleanupTask:清理未完成的分片上传残留块。
|
||
|
|
|
||
|
|
章节来源
|
||
|
|
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
|
||
|
|
- [FileProperties.java](file://crm-file/src/main/java/com/crm/file/config/FileProperties.java)
|
||
|
|
- [SchedulingConfig.java](file://crm-file/src/main/java/com/crm/file/config/SchedulingConfig.java)
|
||
|
|
- [FileApi.java](file://crm-file/src/main/java/com/crm/file/api/FileApi.java)
|
||
|
|
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
- [FileInfoServiceImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileInfoServiceImpl.java)
|
||
|
|
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)
|
||
|
|
- [FileInfo.java](file://crm-file/src/main/java/com/crm/file/domain/entity/FileInfo.java)
|
||
|
|
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
|
||
|
|
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)
|
||
|
|
|
||
|
|
## 架构总览
|
||
|
|
整体采用“控制器-服务-存储”分层设计:控制器负责入参校验与响应封装;服务层编排业务逻辑(含分片上传状态机、权限控制、元数据持久化);存储层通过 MinIO 客户端完成对象存取,并通过数据库记录文件元数据。
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
sequenceDiagram
|
||
|
|
participant C as "客户端"
|
||
|
|
participant Ctrl as "FileController"
|
||
|
|
participant Api as "FileApiImpl"
|
||
|
|
participant DB as "FileInfoMapper"
|
||
|
|
participant MinIO as "MinIO 客户端"
|
||
|
|
participant Preview as "KkFileViewClient"
|
||
|
|
C->>Ctrl : "POST /upload/init"
|
||
|
|
Ctrl->>Api : "initiateMultipart()"
|
||
|
|
Api->>DB : "创建上传会话/记录"
|
||
|
|
Api-->>C : "返回分片ID/策略"
|
||
|
|
loop "分片上传"
|
||
|
|
C->>Ctrl : "POST /upload/chunk"
|
||
|
|
Ctrl->>Api : "uploadChunk()"
|
||
|
|
Api->>MinIO : "上传分片"
|
||
|
|
Api->>DB : "更新分片状态"
|
||
|
|
Api-->>C : "确认分片成功"
|
||
|
|
end
|
||
|
|
C->>Ctrl : "POST /upload/complete"
|
||
|
|
Ctrl->>Api : "completeMultipart()"
|
||
|
|
Api->>MinIO : "合并分片"
|
||
|
|
Api->>DB : "写入文件元数据"
|
||
|
|
Api-->>C : "返回文件信息"
|
||
|
|
C->>Ctrl : "GET /download/{id}"
|
||
|
|
Ctrl->>Api : "download()"
|
||
|
|
Api->>MinIO : "流式下载"
|
||
|
|
Api-->>C : "文件流"
|
||
|
|
C->>Ctrl : "GET /preview/{id}"
|
||
|
|
Ctrl->>Api : "preview()"
|
||
|
|
Api->>Preview : "生成预览链接"
|
||
|
|
Api-->>C : "预览地址"
|
||
|
|
```
|
||
|
|
|
||
|
|
图表来源
|
||
|
|
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
|
||
|
|
- [FileApi.java](file://crm-file/src/main/java/com/crm/file/api/FileApi.java)
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
|
||
|
|
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
|
||
|
|
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)
|
||
|
|
|
||
|
|
## 详细组件分析
|
||
|
|
|
||
|
|
### MinIO 配置与连接
|
||
|
|
- 连接参数
|
||
|
|
- 端点地址、访问密钥、安全协议、连接池、超时时间等由配置类集中管理,便于多环境切换。
|
||
|
|
- 存储空间管理
|
||
|
|
- 支持按业务域划分桶或前缀,结合权限策略限制读写范围。
|
||
|
|
- 访问权限控制
|
||
|
|
- 基于策略的桶/对象级访问控制,结合签名 URL 实现临时访问授权。
|
||
|
|
- 大文件处理策略
|
||
|
|
- 使用分片上传降低内存占用,提升网络容错;服务端维护分片状态,失败重试与幂等合并。
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
classDiagram
|
||
|
|
class MinioConfig {
|
||
|
|
+ "初始化MinIO客户端"
|
||
|
|
+ "获取连接参数"
|
||
|
|
+ "设置超时与重试"
|
||
|
|
}
|
||
|
|
class FileProperties {
|
||
|
|
+ "读取文件配置"
|
||
|
|
+ "分片大小/超时/路径前缀"
|
||
|
|
}
|
||
|
|
class FileApiImpl {
|
||
|
|
+ "分片上传/合并"
|
||
|
|
+ "权限校验"
|
||
|
|
+ "元数据持久化"
|
||
|
|
}
|
||
|
|
MinioConfig <.. FileApiImpl : "被调用"
|
||
|
|
FileProperties <.. FileApiImpl : "被调用"
|
||
|
|
```
|
||
|
|
|
||
|
|
图表来源
|
||
|
|
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
|
||
|
|
- [FileProperties.java](file://crm-file/src/main/java/com/crm/file/config/FileProperties.java)
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
|
||
|
|
章节来源
|
||
|
|
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
|
||
|
|
- [FileProperties.java](file://crm-file/src/main/java/com/crm/file/config/FileProperties.java)
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
|
||
|
|
### 文件上传下载配置
|
||
|
|
- 上传流程
|
||
|
|
- 初始化分片会话、逐片上传、完成后合并为完整对象,并落库元数据。
|
||
|
|
- 下载流程
|
||
|
|
- 根据文件 ID 定位对象,流式返回,避免大文件 OOM。
|
||
|
|
- 预览能力
|
||
|
|
- 通过第三方预览服务生成预览链接,减少服务端渲染压力。
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
flowchart TD
|
||
|
|
Start(["开始"]) --> Init["初始化分片会话"]
|
||
|
|
Init --> ChunkLoop{"是否还有分片?"}
|
||
|
|
ChunkLoop --> |是| Upload["上传分片"]
|
||
|
|
Upload --> UpdateMeta["更新分片状态"]
|
||
|
|
UpdateMeta --> ChunkLoop
|
||
|
|
ChunkLoop --> |否| Complete["合并分片"]
|
||
|
|
Complete --> Persist["持久化文件元数据"]
|
||
|
|
Persist --> Done(["结束"])
|
||
|
|
```
|
||
|
|
|
||
|
|
图表来源
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
- [MultipartInitDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/MultipartInitDTO.java)
|
||
|
|
- [UploadSession.java](file://crm-file/src/main/java/com/crm/file/domain/dto/UploadSession.java)
|
||
|
|
- [FileDownloadDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/FileDownloadDTO.java)
|
||
|
|
- [FileInfoDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/FileInfoDTO.java)
|
||
|
|
|
||
|
|
章节来源
|
||
|
|
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
|
||
|
|
- [FileApi.java](file://crm-file/src/main/java/com/crm/file/api/FileApi.java)
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
- [MultipartInitDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/MultipartInitDTO.java)
|
||
|
|
- [UploadSession.java](file://crm-file/src/main/java/com/crm/file/domain/dto/UploadSession.java)
|
||
|
|
- [FileDownloadDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/FileDownloadDTO.java)
|
||
|
|
- [FileInfoDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/FileInfoDTO.java)
|
||
|
|
|
||
|
|
### 分片上传配置
|
||
|
|
- 分片大小与并发
|
||
|
|
- 通过配置项控制分片大小与并发度,平衡吞吐与内存占用。
|
||
|
|
- 会话与状态
|
||
|
|
- 使用会话对象跟踪分片进度,支持断点续传与失败重试。
|
||
|
|
- 幂等与一致性
|
||
|
|
- 合并前校验分片完整性,确保最终一致。
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
classDiagram
|
||
|
|
class UploadSession {
|
||
|
|
+ "会话ID"
|
||
|
|
+ "分片列表"
|
||
|
|
+ "状态"
|
||
|
|
+ "过期时间"
|
||
|
|
}
|
||
|
|
class MultipartInitDTO {
|
||
|
|
+ "文件名"
|
||
|
|
+ "分片大小"
|
||
|
|
+ "业务标识"
|
||
|
|
}
|
||
|
|
class FileApiImpl {
|
||
|
|
+ "initiateMultipart()"
|
||
|
|
+ "uploadChunk()"
|
||
|
|
+ "completeMultipart()"
|
||
|
|
}
|
||
|
|
UploadSession <.. FileApiImpl : "维护"
|
||
|
|
MultipartInitDTO <.. FileApiImpl : "输入"
|
||
|
|
```
|
||
|
|
|
||
|
|
图表来源
|
||
|
|
- [UploadSession.java](file://crm-file/src/main/java/com/crm/file/domain/dto/UploadSession.java)
|
||
|
|
- [MultipartInitDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/MultipartInitDTO.java)
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
|
||
|
|
章节来源
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
- [UploadSession.java](file://crm-file/src/main/java/com/crm/file/domain/dto/UploadSession.java)
|
||
|
|
- [MultipartInitDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/MultipartInitDTO.java)
|
||
|
|
|
||
|
|
### 定时任务配置
|
||
|
|
- 任务类型
|
||
|
|
- 清理未完成的分片上传残留块,释放存储空间。
|
||
|
|
- 调度策略
|
||
|
|
- 通过调度配置类启用任务,支持 Cron 表达式与线程池隔离。
|
||
|
|
- 执行保障
|
||
|
|
- 任务幂等、异常隔离、日志记录与告警接入。
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
sequenceDiagram
|
||
|
|
participant Scheduler as "SchedulingConfig"
|
||
|
|
participant Task as "OrphanChunkCleanupTask"
|
||
|
|
participant DB as "FileInfoMapper"
|
||
|
|
participant MinIO as "MinIO 客户端"
|
||
|
|
Scheduler->>Task : "触发定时任务"
|
||
|
|
Task->>DB : "查询过期/未完成分片"
|
||
|
|
Task->>MinIO : "删除残留分片"
|
||
|
|
Task->>DB : "清理会话记录"
|
||
|
|
Task-->>Scheduler : "任务完成"
|
||
|
|
```
|
||
|
|
|
||
|
|
图表来源
|
||
|
|
- [SchedulingConfig.java](file://crm-file/src/main/java/com/crm/file/config/SchedulingConfig.java)
|
||
|
|
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)
|
||
|
|
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
|
||
|
|
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
|
||
|
|
|
||
|
|
章节来源
|
||
|
|
- [SchedulingConfig.java](file://crm-file/src/main/java/com/crm/file/config/SchedulingConfig.java)
|
||
|
|
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)
|
||
|
|
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
|
||
|
|
|
||
|
|
### 文件元数据与预览
|
||
|
|
- 元数据模型
|
||
|
|
- 文件基本信息、存储位置、分片状态、访问策略等。
|
||
|
|
- 预览集成
|
||
|
|
- 通过预览客户端生成预览链接,支持多种格式。
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
erDiagram
|
||
|
|
FILE_INFO {
|
||
|
|
bigint id PK
|
||
|
|
string name
|
||
|
|
string bucket
|
||
|
|
string object_key
|
||
|
|
long size
|
||
|
|
string mime_type
|
||
|
|
string status
|
||
|
|
datetime created_at
|
||
|
|
datetime updated_at
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
图表来源
|
||
|
|
- [FileInfo.java](file://crm-file/src/main/java/com/crm/file/domain/entity/FileInfo.java)
|
||
|
|
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
|
||
|
|
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)
|
||
|
|
|
||
|
|
章节来源
|
||
|
|
- [FileInfo.java](file://crm-file/src/main/java/com/crm/file/domain/entity/FileInfo.java)
|
||
|
|
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
|
||
|
|
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)
|
||
|
|
|
||
|
|
### 不同存储后端切换方法
|
||
|
|
- 抽象接口
|
||
|
|
- 以 FileApi 作为统一接口,屏蔽底层存储差异。
|
||
|
|
- 实现替换
|
||
|
|
- 新增其他存储实现类,注入到服务层,通过配置切换。
|
||
|
|
- 配置驱动
|
||
|
|
- 使用 FileProperties 统一管理后端参数,支持多环境差异化配置。
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
classDiagram
|
||
|
|
class FileApi {
|
||
|
|
<<interface>>
|
||
|
|
+ "上传/下载/分片/删除"
|
||
|
|
}
|
||
|
|
class FileApiImpl {
|
||
|
|
+ "MinIO 实现"
|
||
|
|
}
|
||
|
|
class OtherStorageImpl {
|
||
|
|
+ "其他存储实现"
|
||
|
|
}
|
||
|
|
FileApi <|.. FileApiImpl
|
||
|
|
FileApi <|.. OtherStorageImpl
|
||
|
|
```
|
||
|
|
|
||
|
|
图表来源
|
||
|
|
- [FileApi.java](file://crm-file/src/main/java/com/crm/file/api/FileApi.java)
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
|
||
|
|
章节来源
|
||
|
|
- [FileApi.java](file://crm-file/src/main/java/com/crm/file/api/FileApi.java)
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
- [FileProperties.java](file://crm-file/src/main/java/com/crm/file/config/FileProperties.java)
|
||
|
|
|
||
|
|
## 依赖关系分析
|
||
|
|
- 控制器依赖服务接口,服务实现依赖配置与数据访问层。
|
||
|
|
- 定时任务依赖数据访问与存储客户端。
|
||
|
|
- 预览功能依赖外部预览服务客户端。
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
graph LR
|
||
|
|
Controller["FileController"] --> ApiIf["FileApi"]
|
||
|
|
ApiIf --> ApiImpl["FileApiImpl"]
|
||
|
|
ApiImpl --> MinioCfg["MinioConfig"]
|
||
|
|
ApiImpl --> FileProp["FileProperties"]
|
||
|
|
ApiImpl --> Mapper["FileInfoMapper"]
|
||
|
|
ApiImpl --> Preview["KkFileViewClient"]
|
||
|
|
Scheduler["SchedulingConfig"] --> Task["OrphanChunkCleanupTask"]
|
||
|
|
Task --> Mapper
|
||
|
|
Task --> MinioCfg
|
||
|
|
```
|
||
|
|
|
||
|
|
图表来源
|
||
|
|
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
|
||
|
|
- [FileApi.java](file://crm-file/src/main/java/com/crm/file/api/FileApi.java)
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
|
||
|
|
- [FileProperties.java](file://crm-file/src/main/java/com/crm/file/config/FileProperties.java)
|
||
|
|
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
|
||
|
|
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)
|
||
|
|
- [SchedulingConfig.java](file://crm-file/src/main/java/com/crm/file/config/SchedulingConfig.java)
|
||
|
|
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)
|
||
|
|
|
||
|
|
章节来源
|
||
|
|
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
|
||
|
|
- [FileApi.java](file://crm-file/src/main/java/com/crm/file/api/FileApi.java)
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
|
||
|
|
- [FileProperties.java](file://crm-file/src/main/java/com/crm/file/config/FileProperties.java)
|
||
|
|
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
|
||
|
|
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)
|
||
|
|
- [SchedulingConfig.java](file://crm-file/src/main/java/com/crm/file/config/SchedulingConfig.java)
|
||
|
|
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)
|
||
|
|
|
||
|
|
## 性能考虑
|
||
|
|
- 分片大小与并发
|
||
|
|
- 根据网络带宽与服务器内存调整分片大小与并发数,避免拥塞与 OOM。
|
||
|
|
- 流式 I/O
|
||
|
|
- 下载与上传均采用流式处理,减少峰值内存占用。
|
||
|
|
- 连接池与超时
|
||
|
|
- 合理设置 MinIO 客户端连接池、读写超时与重试次数,提升稳定性。
|
||
|
|
- 缓存与预热
|
||
|
|
- 热点文件可引入缓存层(如 Redis)加速访问,注意一致性策略。
|
||
|
|
- 监控与限流
|
||
|
|
- 对上传/下载接口进行速率限制与指标采集,防止资源耗尽。
|
||
|
|
|
||
|
|
[本节为通用性能建议,不直接分析具体文件]
|
||
|
|
|
||
|
|
## 故障排查指南
|
||
|
|
- 连接问题
|
||
|
|
- 检查 MinIO 端点、密钥、协议与网络连通性;查看客户端超时与重试配置。
|
||
|
|
- 分片上传失败
|
||
|
|
- 核对分片顺序与 MD5/ETag;检查会话过期与并发冲突;查看任务清理策略。
|
||
|
|
- 下载异常
|
||
|
|
- 确认对象存在与权限;检查流式下载与响应头设置。
|
||
|
|
- 预览不可用
|
||
|
|
- 验证预览服务可达性与鉴权;检查文件类型与大小限制。
|
||
|
|
- 定时任务未执行
|
||
|
|
- 检查调度配置与线程池;查看任务日志与异常堆栈。
|
||
|
|
|
||
|
|
章节来源
|
||
|
|
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
|
||
|
|
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
|
||
|
|
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)
|
||
|
|
- [SchedulingConfig.java](file://crm-file/src/main/java/com/crm/file/config/SchedulingConfig.java)
|
||
|
|
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)
|
||
|
|
|
||
|
|
## 结论
|
||
|
|
本文件存储子系统以 MinIO 为核心,提供完整的上传下载、分片上传、权限控制与预览能力,并通过定时任务保障资源健康。通过配置驱动与接口抽象,可实现不同存储后端的平滑切换。建议在生产环境完善监控告警、容量规划与备份恢复策略,持续优化性能与稳定性。
|
||
|
|
|
||
|
|
[本节为总结性内容,不直接分析具体文件]
|
||
|
|
|
||
|
|
## 附录
|
||
|
|
- 配置项清单
|
||
|
|
- MinIO 连接参数:端点、密钥、协议、超时、连接池等。
|
||
|
|
- 文件策略:桶名、路径前缀、分片大小、并发度、访问策略。
|
||
|
|
- 调度任务:Cron 表达式、线程池大小、清理阈值。
|
||
|
|
- 容量监控与备份恢复
|
||
|
|
- 定期统计桶/对象数量与容量趋势;制定快照与异地备份策略;演练恢复流程。
|
||
|
|
- 权限与安全
|
||
|
|
- 最小权限原则;签名 URL 短期有效;敏感文件加密存储。
|
||
|
|
|
||
|
|
[本节为补充说明,不直接分析具体文件]
|