【.net core使用minio大文件分片上传】.net core使用minio大文件分片上传以及断点续传、秒传思路

版本：.net core 7
需求：net限制了上传的大小，只能上传25M上下的文件，如果上传一个八十多兆的文件，swagger接口报错，如果前端调用上传接口，会报CORS跨域错误，这篇文章介绍怎么使用分片的方式上传到minio后合并。

1.安装AWSSDK.S3的SDK

miniio 的分片上传封装，采用了AWSSDK.S3的SDK，该SDK是兼容了亚马逊s3的接口api，而minio是采用亚马逊s3的api模式

2.带上我写的后端MinioHelper.cs文件的封装方法

ConfigConstant.MinioBucketName是minio的存储桶名称，这里示例值是micro-element
ConfigConstant.MinioUserName是minio的AccessKey，值以你自己minio服务器的AccessKey为准
ConfigConstant.MinioPassWord是minio的SecretKey，值以你自己minio服务器的SecretKey为准
ConfigConstant.MinioAddressIp是minio的地址，需要带http://ip:port，比如http://192.168.110.11/

/// <summary>
/// miniio 的分片上传封装，采用了AWSSDK.S3的SDK，该SDK是兼容了亚马逊s3的接口api，而minio是采用亚马逊s3的api模式
/// </summary>
public class MinioHelper
{private  AmazonS3Client amazonS3Client;public MinioHelper(){AmazonS3Config config = new AmazonS3Config(){ServiceURL = ConfigConstant.MinioAddressIp  //地址采用服务器minio的http://ip:port};amazonS3Client = new AmazonS3Client(ConfigConstant.MinioUserName, ConfigConstant.MinioPassWord, config);//minio的账号密码}/// <summary>/// 初始化文件获取uploadid/// </summary>/// <param name="BucketName">桶名称</param>/// <param name="KeyName">存储文件的桶路径</param>/// <returns>返回uploadID</returns>public async   Task<string> InitMultipartChunkUploadFile(string BucketName,string KeyName) {InitiateMultipartUploadRequest initRequest=  new InitiateMultipartUploadRequest{BucketName = BucketName,Key = KeyName,};InitiateMultipartUploadResponse initResponse = await amazonS3Client.InitiateMultipartUploadAsync(initRequest);return  initResponse.UploadId;}/// <summary>/// 上传分片，这是看minio web控制台是看不出来分片文件的，需要去minio存储文件夹下的.minio.sys/multipart查看/// </summary>/// <param name="BucketName">桶名称</param>/// <param name="KeyName">存储文件的桶路径</param>/// <param name="UploadId">uploadID</param>/// <param name="ChunkCount">当前第几片</param>/// <param name="PartSize">当前分片大小</param>/// <param name="ChunkFile">分片文件</param>/// <returns>返回etag和当前第几片</returns>public async  Task<PartETag> ChunkUploadAsync(string BucketName, string KeyName,string UploadId, int ChunkCount, long PartSize, Stream ChunkFile) {UploadPartRequest uploadRequest = new UploadPartRequest();uploadRequest.BucketName = BucketName;uploadRequest.Key = KeyName;uploadRequest.UploadId = UploadId;uploadRequest.PartNumber = ChunkCount;uploadRequest.InputStream = ChunkFile;uploadRequest.PartSize = PartSize;//进行分片上传UploadPartResponse up1Response = await amazonS3Client.UploadPartAsync(uploadRequest);return  new PartETag { ETag = up1Response.ETag, PartNumber = ChunkCount };}/// <summary>/// 合并分片为整个文件，合并后minio存储的文件夹里.minio.sys/multipart下该文件会删除/// </summary>/// <param name="BucketName">桶名称</param>/// <param name="KeyName">存储文件的桶路径</param>/// <param name="UploadId">uploadID</param>/// <param name="partETags">当前第几片</param>/// <returns>返回存储的Key属性字段的相对路径(location是绝对路径，该属性不可取，如果服务器迁移会使人崩溃！)</returns>public async  Task<CompleteMultipartUploadResponse> CompleteMultipartUploadFile(string BucketName, string KeyName, string UploadId, List<PartETag> partETags) {CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest{BucketName = BucketName,Key = KeyName,UploadId = UploadId,PartETags = partETags,};CompleteMultipartUploadResponse compResponse = await amazonS3Client.CompleteMultipartUploadAsync(compRequest);return compResponse;}/// <summary>/// 查询该uploadID已上传了几个分片详细信息/// </summary>/// <param name="BucketName">桶名称</param>/// <param name="KeyName">存储文件的桶路径</param>/// <param name="UploadId">uploadID</param>/// <returns>只需要返回值数组下每个对象里的eTag和partNumber</returns>public async  Task<List<PartDetail>> ListPartsUploadInfo(string BucketName, string KeyName, string UploadId) {ListPartsRequest listPartRequest = new ListPartsRequest{BucketName = BucketName,Key = KeyName,UploadId = UploadId};ListPartsResponse listPartResponse = await amazonS3Client.ListPartsAsync(listPartRequest);return listPartResponse.Parts;}
}

3.在需要使用的地方引入该项目

4.申明MinioHelper文件的构造函数

        private MinioHelper minioHelper;/// <summary>/// /// </summary>public minioUploadController(){minioHelper = new MinioHelper();}

5.初始化以及上传分片接口

 /// <summary>/// 查询上传分片列表/// </summary>/// <param name="files"></param>/// <returns></returns>[HttpPost("/file/ListPartsFile")]public async Task<dynamic> ListPartsFile(string Name, string UploadId = ""){List<PartDetail> listPartResponse =new List<PartDetail>();try{listPartResponse = await minioHelper.ListPartsUploadInfo(ConfigConstant.MinioBucketName, "micro-elementtest/" + Name, UploadId);Console.WriteLine("查询上传分片列表" + JsonConvert.SerializeObject(listPartResponse));}catch (MinioException e){Log.Error("查询上传分片列表错误: {0}", e.Message);}return Ok(new {  UploadId, listPartResponse });}/// <summary>/// 合并分片/// </summary>/// <param name="files"></param>/// <returns></returns>[HttpPost("/file/ChunkMultipartUpload")]public async Task<dynamic> ChunkUploadFile( string Name, string UploadId = "", List<PartETag> partETagList=null ){CompleteMultipartUploadResponse compResponse=new CompleteMultipartUploadResponse();try{// Complete the multipart upload  分片上传完后合并compResponse = await minioHelper.CompleteMultipartUploadFile(ConfigConstant.MinioBucketName, "micro-elementtest/" + Name, UploadId, partETagList);Console.WriteLine("分片上传"+ JsonConvert.SerializeObject(compResponse.Key));}catch (MinioException e){Log.Error("文件上传错误: {0}", e.Message);}return Ok(new {  compResponse });}/// <summary>/// 上传分片/// </summary>/// <param name="files"></param>/// <returns></returns>[HttpPost("/file/ChunkUploadFile")]public async Task<dynamic> ChunkUploadFile(IFormFile files, [FromForm] string ID, [FromForm]  string Name, [FromForm] long Size, [FromForm] long partSize,  [FromForm] int ChunkCount, [FromForm] string UploadId = ""){List<PartETag> partETagList = new List<PartETag>();try{// Define input streamStream inputStream = files.OpenReadStream(); //Create13MBDataStream();if (string.IsNullOrWhiteSpace(UploadId)){//初始化分片上传，得到UploadIdUploadId = await minioHelper.InitMultipartChunkUploadFile(ConfigConstant.MinioBucketName, "micro-elementtest/" + Name);}PartETag partETag = await minioHelper.ChunkUploadAsync(ConfigConstant.MinioBucketName, "micro-elementtest/" + Name, UploadId, ChunkCount, partSize, inputStream);partETagList.Add(partETag);}catch (MinioException e){Log.Error("文件上传错误: {0}", e.Message);}return Ok(new { UploadId, partETagList, Name });}

6.前端页面，采用的vue2+element-ui的框架

<template><div><el-upload class="upload-demo" drag action="" multiple :http-request="handHttpRequest"><i class="el-icon-upload"></i><div class="el-upload__text">将文件拖到此处，或<em>点击上传</em></div><div class="el-upload__tip" slot="tip">只能上传jpg/png文件，且不超过500kb</div></el-upload></div>
</template><script>
import { ChunkUploadFileApi } from '@/api/upload'
export default {name: 'uploadPage',data() {return {uploadId: '',ETag1: '',ETag2: '',}},methods: {handtap() {},async handHttpRequest(params) {var chunkSize = 5 * 1024 * 1024 // 1MB一片const name = params.file.namelet UploadId = ''const uploadNextChunk = async (i) => {const file = this.getChunkInfo(params.file, i, chunkSize)console.log(file)const partSize = file.chunk.sizeconst ChunkCount = i+1const res = await ChunkUploadFileApi(file.chunk, '123123123123123sa', name, partSize, ChunkCount, 123154545, UploadId)console.log(res)UploadId = res.uploadIdif (i < 1) {// 仅上传两片，根据需求修改await uploadNextChunk(i + 1)}}await uploadNextChunk(0)},getChunkInfo(file, currentChunk, chunkSize) {var start = currentChunk * chunkSizevar end = Math.min(file.size, start + chunkSize)var chunk = file.slice(start, end)return {start,end,chunk,}},},
}
</script><style></style>

upload文件内容为

import request from '@/utils/request'//这个是request
import base from '@/api/base'//这个是我自己文件的api前缀const noteApi = {ChunkUploadFile: base.outApi + '/file/ChunkUploadFile', //
}/*** 分片测试* @returns*/
export const ChunkUploadFileApi = (file, hash, name, partSize, ChunkCount, size, UploadId) => {const formData = new FormData();formData.append("files", file);formData.append("Name", name);formData.append("ID", hash);formData.append("PartSize", partSize);formData.append("ChunkCount", ChunkCount);formData.append("Size", size);formData.append("UploadId", UploadId);return request({url: noteApi.ChunkUploadFile,method: "post",data: formData,headersType: "multipart/form-data",});};

7.在前端页面上传分片文件

我这里分片是以5MB为一个分片，超过5MB才使用分片的功能

通过接口看到上传成功了

8.通过ListPartsFile方法查询已上传的分片

上传的分片这时候在minio的web控制台是看不到的，需要去minio的存储文件夹里查看路径/data/.minio.sys/multipart

9.合并分片

合并后在minio桶里可以看到文件了

10.注意点：

ConfigConstant.MinioBucketName是minio的存储桶名称，这里示例值是micro-element
ConfigConstant.MinioUserName是minio的AccessKey，值以你自己minio服务器的AccessKey为准
ConfigConstant.MinioPassWord是minio的SecretKey，值以你自己minio服务器的SecretKey为准
ConfigConstant.MinioAddressIp是minio的地址，需要带http://ip:port，比如http://192.168.110.11/

1.文件只有大于5MB才能使用分片，如果小于5MB，则不可以使用分片功能

2.文件分片的流程是：先初始化通过InitMultipartChunkUploadFile接口获取uploadID==>通过ChunkUploadAsync接口上传分片的文件（只有最后一片允许小于5MB，前面的分片必须要等于5MB）==>通过CompleteMultipartUploadFile接口合并分片

3.上传分片后还没有合并前，分片文件在桶里不可见，因为分片文件在你的minio存储路径的/minio/data/.minio.sys/multipart路径下

4.合并后文件建议取CompleteMultipartUploadFile接口返回值的key相对路径，如果取绝对路径location的话，以后迁移服务器会使用崩溃~

5.现在已经有了分片的教程，后续的大文件秒传、断点续传可以根据上面的封装接口举一反三，很简单~
（秒传就是判断文件的md5和size大小合并查询数据库是否有重复，如果有，则直接返回文件信息）
（断点续传就是判断文件的md5和size在数据库是否存在并且未上传完成，则根据已上传的文件去判断还有哪些文件没有上传，直接续传即可）