crm-backend-matt/.qoder/repowiki/zh/content/架构设计/存储架构设计.md

# 存储架构设计

<cite>
**本文引用的文件**   
- [CrmAppApplication.java](file://crm-app/src/main/java/com/crm/app/CrmAppApplication.java)
- [application.yml](file://crm-app/src/main/resources/application.yml)
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
- [FileProperties.java](file://crm-file/src/main/java/com/crm/file/config/FileProperties.java)
- [SchedulingConfig.java](file://crm-file/src/main/java/com/crm/file/config/SchedulingConfig.java)
- [FileConstants.java](file://crm-file/src/main/java/com/crm/file/constant/FileConstants.java)
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
- [FileInfo.java](file://crm-file/src/main/java/com/crm/file/domain/entity/FileInfo.java)
- [MultipartInitDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/MultipartInitDTO.java)
- [UploadSession.java](file://crm-file/src/main/java/com/crm/file/domain/dto/UploadSession.java)
- [FileDownloadDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/FileDownloadDTO.java)
- [FileInfoDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/FileInfoDTO.java)
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
- [IFileInfoService.java](file://crm-file/src/main/java/com/crm/file/service/IFileInfoService.java)
- [FileInfoServiceImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileInfoServiceImpl.java)
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)
- [BaseEntity.java](file://crm-base/src/main/java/com/crm/base/domain/entity/BaseEntity.java)
- [OwnedEntity.java](file://crm-base/src/main/java/com/crm/base/domain/entity/OwnedEntity.java)
</cite>

## 目录
1. [简介](#简介)
2. [项目结构](#项目结构)
3. [核心组件](#核心组件)
4. [架构总览](#架构总览)
5. [详细组件分析](#详细组件分析)
6. [依赖关系分析](#依赖关系分析)
7. [性能考量](#性能考量)
8. [故障排查指南](#故障排查指南)
9. [结论](#结论)
10. [附录](#附录)

## 简介
本文件面向CRM系统的存储架构设计与实现，重点围绕以下目标展开：
- 基于MinIO的对象存储服务集成与配置管理
- 大文件分片上传、断点续传与文件去重机制
- 在线预览能力（通过KkFileView客户端）
- FileInfo实体的元数据管理与生命周期
- 安全访问控制、存储空间监控与备份恢复策略

文档同时为文件服务开发者与运维人员提供端到端的技术参考。

## 项目结构
系统采用多模块组织，存储相关能力集中在crm-file模块，基础能力在crm-base模块，应用入口在crm-app模块。关键目录与职责如下：
- crm-app: 应用启动与全局配置
- crm-file: 文件服务核心，包含控制器、配置、领域模型、服务实现、定时任务等
- crm-base: 通用实体、工具与安全上下文

```mermaid
graph TB
subgraph "应用层"
APP["CrmAppApplication<br/>应用启动"]
end
subgraph "文件服务(crm-file)"
CTRL["FileController<br/>文件接口"]
CFG["MinioConfig / FileProperties<br/>配置"]
SVC["FileInfoServiceImpl / FileApiImpl<br/>业务逻辑"]
DTO["DTOs: MultipartInitDTO, UploadSession,<br/>FileDownloadDTO, FileInfoDTO"]
ENT["FileInfo<br/>元数据实体"]
MAPPER["FileInfoMapper<br/>持久化"]
KK["KkFileViewClient<br/>预览客户端"]
TASK["OrphanChunkCleanupTask<br/>碎片清理"]
end
subgraph "基础设施"
MINIO["MinIO对象存储"]
DB["数据库(MyBatis-Plus)"]
KKV["KkFileView预览服务"]
end
APP --> CTRL
CTRL --> SVC
SVC --> CFG
SVC --> ENT
SVC --> MAPPER
SVC --> MINIO
SVC --> KK
TASK --> MINIO
MAPPER --> DB
```

图表来源
- [CrmAppApplication.java](file://crm-app/src/main/java/com/crm/app/CrmAppApplication.java)
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
- [FileProperties.java](file://crm-file/src/main/java/com/crm/file/config/FileProperties.java)
- [FileInfoServiceImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileInfoServiceImpl.java)
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
- [FileInfo.java](file://crm-file/src/main/java/com/crm/file/domain/entity/FileInfo.java)
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)

章节来源
- [CrmAppApplication.java](file://crm-app/src/main/java/com/crm/app/CrmAppApplication.java)
- [application.yml](file://crm-app/src/main/resources/application.yml)

## 核心组件
- MinioConfig: 封装MinIO客户端初始化与连接参数，集中管理Endpoint、AccessKey、SecretKey、Bucket等配置项
- FileProperties: 文件服务业务配置，如分片大小、超时、重试、预览服务地址等
- FileController: 对外暴露文件上传、分片、合并、下载、预览等REST接口
- FileInfoServiceImpl / FileApiImpl: 分片上传流程编排、断点续传状态管理、去重校验、下载与预览协调
- KkFileViewClient: 调用外部KkFileView服务进行在线预览
- FileInfo: 文件元数据实体，记录文件名、路径、大小、类型、哈希、版本、权限等
- FileInfoMapper: 元数据持久化操作
- OrphanChunkCleanupTask: 定时清理未完成的分片，释放空间

章节来源
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
- [FileProperties.java](file://crm-file/src/main/java/com/crm/file/config/FileProperties.java)
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
- [FileInfoServiceImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileInfoServiceImpl.java)
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)
- [FileInfo.java](file://crm-file/src/main/java/com/crm/file/domain/entity/FileInfo.java)
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)

## 架构总览
整体架构围绕“对象存储+元数据+预览服务”的解耦模式构建：
- 上传链路：客户端→FileController→FileApiImpl/FileInfoServiceImpl→MinIO（分片直传或服务端聚合）
- 元数据链路：FileInfo实体→MyBatis-Plus持久化
- 预览链路：FileController→KkFileViewClient→KkFileView服务→返回预览页面或图片
- 清理链路：OrphanChunkCleanupTask定期扫描并删除孤儿分片

```mermaid
sequenceDiagram
participant Client as "客户端"
participant Controller as "FileController"
participant Service as "FileInfoServiceImpl"
participant API as "FileApiImpl"
participant MinIO as "MinIO对象存储"
participant DB as "数据库"
participant Preview as "KkFileViewClient"
Client->>Controller : "发起分片上传(初始化)"
Controller->>Service : "创建上传会话/分配分片策略"
Service->>DB : "持久化元数据(初始状态)"
loop 多次请求
Client->>Controller : "上传分片(chunkIndex, data)"
Controller->>API : "校验分片/幂等处理"
API->>MinIO : "上传分片到临时目录"
API-->>Controller : "分片完成标记"
end
Client->>Controller : "合并分片"
Controller->>Service : "触发合并与完整性校验"
Service->>MinIO : "合并为完整对象"
Service->>DB : "更新元数据(大小/哈希/状态)"
Client->>Controller : "获取下载链接"
Controller->>Service : "生成签名URL/鉴权"
Service-->>Client : "返回下载链接"
Client->>Controller : "请求在线预览"
Controller->>Preview : "转发预览请求"
Preview-->>Client : "返回预览结果"
```

图表来源
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
- [FileInfoServiceImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileInfoServiceImpl.java)
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)

## 详细组件分析

### MinIO集成与配置
- MinioConfig负责MinIO客户端的初始化与连接参数注入，包括Endpoint、AccessKey、SecretKey、Bucket名称、是否启用SSL等
- FileProperties定义文件服务业务参数，如分片大小、最大分片数、超时时间、重试次数、预览服务地址等
- 建议将敏感配置放入环境变量或密钥管理服务，避免硬编码

章节来源
- [MinioConfig.java](file://crm-file/src/main/java/com/crm/file/config/MinioConfig.java)
- [FileProperties.java](file://crm-file/src/main/java/com/crm/file/config/FileProperties.java)

### 文件元数据实体与持久化
- FileInfo实体承载文件元数据，通常包含文件名、原始名、扩展名、存储路径、大小、MIME类型、内容哈希、版本、权限标识、创建/更新时间等
- BaseEntity与OwnedEntity提供通用字段（如ID、审计字段、数据所有权）
- FileInfoMapper提供CRUD与查询能力，支持按哈希去重、按所有者过滤、分页检索等

```mermaid
classDiagram
class BaseEntity {
+id
+createTime
+updateTime
}
class OwnedEntity {
+ownerId
+tenantId
}
class FileInfo {
+fileName
+originalName
+extension
+storagePath
+size
+mimeType
+contentHash
+version
+status
+permissions
+metadata
}
class FileInfoMapper {
+insert(entity)
+updateById(entity)
+selectByHash(hash)
+selectByOwner(ownerId)
+pageQuery(param)
}
BaseEntity <|-- OwnedEntity
OwnedEntity <|-- FileInfo
FileInfo --> FileInfoMapper : "使用"
```

图表来源
- [FileInfo.java](file://crm-file/src/main/java/com/crm/file/domain/entity/FileInfo.java)
- [BaseEntity.java](file://crm-base/src/main/java/com/crm/base/domain/entity/BaseEntity.java)
- [OwnedEntity.java](file://crm-base/src/main/java/com/crm/base/domain/entity/OwnedEntity.java)
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)

章节来源
- [FileInfo.java](file://crm-file/src/main/java/com/crm/file/domain/entity/FileInfo.java)
- [BaseEntity.java](file://crm-base/src/main/java/com/crm/base/domain/entity/BaseEntity.java)
- [OwnedEntity.java](file://crm-base/src/main/java/com/crm/base/domain/entity/OwnedEntity.java)
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)

### 分片上传与断点续传
- 分片策略：根据FileProperties中的分片大小计算总分片数；每个分片携带chunkIndex与uploadId用于幂等与合并
- 断点续传：上传会话UploadSession持久化分片状态，支持失败重试与增量合并
- 去重机制：基于内容哈希(contentHash)判断是否已存在相同文件，命中则直接复用已有对象并返回现有元数据
- 合并流程：所有分片上传完成后，服务端执行合并，校验完整性后更新元数据状态

```mermaid
flowchart TD
Start(["开始"]) --> Init["初始化上传会话<br/>分配uploadId/分片策略"]
Init --> CheckDup{"是否存在相同哈希文件?"}
CheckDup --> |是| ReturnExisting["返回现有文件元数据"]
CheckDup --> |否| UploadChunks["循环上传分片"]
UploadChunks --> ChunkState["记录分片状态/进度"]
ChunkState --> AllUploaded{"所有分片完成?"}
AllUploaded --> |否| Resume["断点续传/重试"]
Resume --> UploadChunks
AllUploaded --> |是| Merge["合并分片为完整对象"]
Merge --> Verify["完整性校验/哈希比对"]
Verify --> UpdateMeta["更新元数据/状态"]
UpdateMeta --> Done(["结束"])
ReturnExisting --> Done
```

图表来源
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
- [FileInfoServiceImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileInfoServiceImpl.java)
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
- [MultipartInitDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/MultipartInitDTO.java)
- [UploadSession.java](file://crm-file/src/main/java/com/crm/file/domain/dto/UploadSession.java)

章节来源
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
- [FileInfoServiceImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileInfoServiceImpl.java)
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
- [MultipartInitDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/MultipartInitDTO.java)
- [UploadSession.java](file://crm-file/src/main/java/com/crm/file/domain/dto/UploadSession.java)

### 在线预览功能
- 通过KkFileViewClient调用外部KkFileView服务，传入文件路径或URL，返回预览页面或图片
- 支持常见办公文档、PDF、图片等格式；对不支持格式可降级为下载
- 预览前需校验文件权限与有效性，确保仅授权用户可访问

```mermaid
sequenceDiagram
participant Client as "客户端"
participant Controller as "FileController"
participant Service as "FileInfoServiceImpl"
participant Preview as "KkFileViewClient"
participant KKV as "KkFileView服务"
Client->>Controller : "请求预览(fileId)"
Controller->>Service : "校验权限/有效性"
Service->>Preview : "构造预览请求(文件路径/URL)"
Preview->>KKV : "转发预览请求"
KKV-->>Preview : "返回预览结果"
Preview-->>Service : "解析响应"
Service-->>Controller : "返回预览链接/数据"
Controller-->>Client : "展示预览"
```

图表来源
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
- [FileInfoServiceImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileInfoServiceImpl.java)
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)

章节来源
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)

### 下载与访问控制
- 下载接口支持签名URL或流式下载，结合权限校验防止越权访问
- 支持按租户/所有者维度隔离，确保数据安全
- 下载DTO(FileDownloadDTO)封装必要参数，便于统一校验与日志记录

章节来源
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
- [FileDownloadDTO.java](file://crm-file/src/main/java/com/crm/file/domain/dto/FileDownloadDTO.java)

### 孤儿分片清理任务
- OrphanChunkCleanupTask定时扫描未完成的分片会话，清理过期分片以释放MinIO空间
- 清理策略：超过阈值时间的会话视为孤儿，删除对应分片并回收资源

章节来源
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)
- [SchedulingConfig.java](file://crm-file/src/main/java/com/crm/file/config/SchedulingConfig.java)

## 依赖关系分析
- 控制器依赖服务层，服务层依赖MinIO客户端与数据库持久化
- 预览能力通过HTTP客户端与外部服务交互，具备容错与重试策略
- 定时任务独立运行，不阻塞主业务流程

```mermaid
graph LR
Controller["FileController"] --> Service["FileInfoServiceImpl"]
Service --> API["FileApiImpl"]
Service --> Mapper["FileInfoMapper"]
Service --> MinIO["MinIO客户端"]
Service --> Preview["KkFileViewClient"]
Task["OrphanChunkCleanupTask"] --> MinIO
Mapper --> DB["数据库"]
```

图表来源
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
- [FileInfoServiceImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileInfoServiceImpl.java)
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)

章节来源
- [FileController.java](file://crm-file/src/main/java/com/crm/file/controller/FileController.java)
- [FileInfoServiceImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileInfoServiceImpl.java)
- [FileApiImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileApiImpl.java)
- [FileInfoMapper.java](file://crm-file/src/main/java/com/crm/file/mapper/FileInfoMapper.java)
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)

## 性能考量
- 分片大小调优：根据网络带宽与服务器内存调整分片大小，平衡并发与内存占用
- 并发上传：合理设置线程池与限流，避免对MinIO造成压力
- 缓存策略：对热点文件的元数据与签名URL进行短期缓存，降低重复计算
- 预览优化：对常用格式预渲染或缓存预览结果，减少外部服务调用延迟
- 存储分层：冷热数据分离，归档历史文件至低成本存储

[本节为通用指导，无需引用具体文件]

## 故障排查指南
- 上传失败：检查分片状态、网络超时、MinIO连接参数与权限
- 合并异常：核对分片完整性、哈希比对结果与临时目录权限
- 预览失败：确认KkFileView服务可达、文件格式支持与鉴权参数正确
- 空间不足：监控MinIO桶容量，及时清理孤儿分片与过期文件
- 权限问题：校验用户角色、数据所有权与访问控制策略

章节来源
- [FileInfoServiceImpl.java](file://crm-file/src/main/java/com/crm/file/service/impl/FileInfoServiceImpl.java)
- [KkFileViewClient.java](file://crm-file/src/main/java/com/crm/file/service/KkFileViewClient.java)
- [OrphanChunkCleanupTask.java](file://crm-file/src/main/java/com/crm/file/task/OrphanChunkCleanupTask.java)

## 结论
本存储架构以MinIO为核心对象存储，结合分片上传、断点续传与去重机制，满足大文件高效可靠传输需求；通过KkFileView实现在线预览，提升用户体验；FileInfo实体统一管理元数据，配合权限控制与定时清理任务，保障系统稳定性与安全性。该方案为CRM系统的文件服务提供了可扩展、高性能且易维护的存储解决方案。

[本节为总结性内容，无需引用具体文件]

## 附录
- 文件生命周期管理流程：上传→分片→合并→校验→元数据更新→可用→下载/预览→归档/清理
- 安全访问控制：基于用户身份与数据所有权进行细粒度权限校验
- 存储空间监控：定期统计桶容量与分片数量，设置告警阈值
- 备份恢复策略：定期快照MinIO桶数据，跨地域复制，支持灾难恢复

[本节为概念性说明，无需引用具体文件]