通义千问模型微调——swift框架

1.创建环境

服务器CUDA Version: 12.2

conda create -n lora_qwen python=3.10 -y 
conda activate lora_qwen 
conda install pytorch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 pytorch-cuda=12.1 -c pytorch -c nvidia -y

1.1环境搭建

本文使用swift进行微调，所以先下载swift，以及一些必要的packages

git clone https://github.com/modelscope/ms-swift.git
pip install transformers==4.49.0 
pip install pyav qwen_vl_utils 
pip install numpy==1.22.4 
pip install modelscope

1.2模型下载

使用modelscope下载指定模型，其中：

--model表示模型名称，可在modelscope官网找到

--local_dir代表模型下载地址

运行下面的命令，模型会下载到：./Qwen/Qwen2.5-VL-7B-Instruct目录下

modelscope download --model Qwen/Qwen2.5-VL-7B-Instruct --local_dir ./

下面脚本用于和模型进行对话，可以简单测试一下模型是否能够使用

CUDA_VISIBLE_DEVICES=1 swift infer --model_type qwen2_5_vl --ckpt_dir ./Qwen/Qwen2.5-VL-7B-Instruct

1.3数据集准备

下方是数据集格式，保存类型为.jsonl

[{"query": "OCR一下<image>","response": "朵拉童衣","images": ["datasets/lora_qwen/train/billboard_00001_010_朵拉童衣.jpg"]},{"query": "OCR一下<image>","response": "童衣雜貨舖","images": ["datasets/lora_qwen/train/billboard_00002_010_童衣雜貨舖.jpg"]},...
]

2.微调

2.1采用LoRA进行微调

对文件夹中之前下载的ms-swift-main/examples/train/multimodal/ocr.sh进行修改

# 20GB
CUDA_VISIBLE_DEVICES=0,1 \
MAX_PIXELS=1003520 \
swift sft \--model ./Qwen/Qwen2.5-VL-7B-Instruct \--model_type qwen2_5_vl \--dataset ./datatsets/train.jsonl \--val_dataset ./datatsets/val.jsonl \--train_type lora \--torch_dtype bfloat16 \--num_train_epochs 100 \--per_device_train_batch_size 1 \--per_device_eval_batch_size 1 \--learning_rate 1e-4 \--lora_rank 64 \--lora_alpha 64 \--target_modules all-linear \--freeze_vit true \--gradient_accumulation_steps 16 \--eval_steps 50 \--save_steps 50 \--save_total_limit 10 \--logging_steps 5 \--max_length 2048 \--output_dir output \--warmup_ratio 0.05 \--dataloader_num_workers 4

常用参数解释：

--model：原模型的权重地址--dataset：训练集的数据地址--val_dataset：验证集的数据地址--train_type：全参数训练(full) 或 LoRA微调训练(lora)--num_train_epochs：总共要训练的轮数--per_device_train_batch_size：训练阶段batchsize大小，根据显存大小来设置--per_device_eval_batch_size：验证阶段batchsize大小，根据显存大小来设置--learning_rate：学习率，一般设为0.0001或0.00001--target_modules：需要做微调的目标模块，all-linear表示所有的线形层，也就是Attention和FeedForward层--freeze_vit：一般设为true，不微调视觉编码器，只微调LLM部分

2.2使用Transformer进行推理

import os
import re
import torch
from PIL import Imagefrom datasets import Dataset
from modelscope import AutoTokenizer
from peft import LoraConfig, TaskType, get_peft_model, PeftModel
from transformers import (AutoProcessor,Qwen2_5_VLForConditionalGeneration,Trainer, TrainingArguments,Seq2SeqTrainer, Seq2SeqTrainingArguments, DataCollatorForSeq2Seq,
)
from qwen_vl_utils import process_vision_inforewrite_print = print
def print(save_txt, *arg, **kwargs):rewrite_print(*arg, **kwargs)rewrite_print(*arg, **kwargs, file=open(save_txt, "a+", encoding="utf-8"))def process_func(model, img_path, input_content):messages = [{"role": "user","content": [{"type": "image", "image": img_path},{"type": "text", "text": input_content},],}]text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)image_inputs, video_inputs = process_vision_info(messages)inputs = processor(text=[text],images=image_inputs,videos=video_inputs,padding=True,return_tensors="pt",)generated_ids = model.generate(**inputs, max_new_tokens=512)generated_ids_trimmed = [out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)]output_text = processor.batch_decode(generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False)print(save_txt_path, img_path)print(save_txt_path, output_text[0])print(save_txt_path, '\n')def get_lora_model(model_path, lora_model_path):model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model_path, device_map="auto", torch_dtype=torch.bfloat16)model.enable_input_require_grads()config = LoraConfig(task_type=TaskType.CAUSAL_LM,target_modules="model\..*layers\.\d+\.(self_attn\.(q_proj|k_proj|v_proj|o_proj)|mlp\.(gate_proj|up_proj|down_proj))",inference_mode=True,r=64,lora_alpha=64,lora_dropout=0.05,bias="none",)peft_model = PeftModel.from_pretrained(model, model_id=lora_model_path, config=config)return peft_modelif __name__ == '__main__':save_txt_path = 'log.txt'model_path = "./Qwen2.5-VL-7B-Instruct"lora_model_path = "./output/v2-20250228-202446/checkpoint-900"lora_model = get_lora_model(model_path, lora_model_path)processor = AutoProcessor.from_pretrained(model_path)img_path = "图片路径"prompt = "OCR一下"process_func(lora_model, img_path, prompt)