大语言模型数据类型与环境配置

前言

简单给出环境安装与数据类型及vscode运行配置，其中vscode运行配置是便于我们调试代码。

一、环境安装

直接下面一句话，可实现环境安装，如下：

pip install -r requirements.txt  

而requirements.txt内容如下：

peft==0.7.1
torch==2.1.1
transformers==4.40.0
bitsandbytes
datasets
safetensors
scikit-learn
deepspeed

二、大语言模型数据类型

大语言模型任务很多是和数据挂钩，给什么样的数据，便会训练成不同任务模型，我给出网上一些样列，这些数据基本来源huggingface，给出部分样例供参考！

1、基本文本指令数据类型

2、数学指令数据类型

3、几何图形指令数据类型

4、多模态指令数据类型

5、翻译指令数据类型

三、vscode配置

我先使用python与torchrun方法来运行训练脚本，理论上需改写成一个sh脚本，这些都比较简单。这里，是为了自己搭建大语言模型llama3，为了调试运行正确与否，我先直接给出vscode配置内容，依然是launch.json配置，如下：

{"version": "0.2.0","configurations": [{"name": "train_llama3","type": "python","request": "launch","python": "/home/miniconda3/envs/llama3/bin/python", // 指定python解释器"program": "/language_model/Chinese-LLaMA-Alpaca-3-main/llama3_model/main.py","console": "integratedTerminal","justMyCode": false, "args": [ "--model_name_or_path","/language_model/Chinese-LLaMA-Alpaca-3-main/llama3_8b_weight","--tokenizer_name_or_path","/language_model/Chinese-LLaMA-Alpaca-3-main/llama3_8b_weight","--dataset_dir","/language_model/Chinese-LLaMA-Alpaca-3-main/data_math/school_math.json","--output_dir","./output_dir",// trainer的参数"--data_cache_dir","temp_data_cache_dir","--torch_dtype","bfloat16","--per_device_train_batch_size","1","--do_train","--low_cpu_mem_usage","--num_train_epochs","1","--lr_scheduler_type", "cosine","--learning_rate","1e-4","--warmup_ratio","0.05","--weight_decay","0.01","--logging_strategy","steps", "--logging_steps","10","--save_strategy","steps","--save_total_limit","3","--save_steps","240","--gradient_accumulation_steps","8","--preprocessing_num_workers","8","--tune_lm_head","False","--use_lora","True","--lora_rank","64","--lora_alpha","128","--lora_trainable","q_proj,v_proj,k_proj,o_proj,gate_proj,down_proj,up_proj","--lora_dropout","0.05",],"env": {"CUDA_VISIBLE_DEVICES": "0"},},}