避免辣鸡网站隐藏后文，先上结论
| model_name | memory |
|---|---|
| qwen2-7b-int8 | 11.6G |
| qwen2-7b-int4 | 8.5G |
| qwen2-1.5b | 4.2G |
| qwen2-1.5b-int8 | 3G |
| qwen2-1.5b-int4 | 2.5G |
btw: ollama部署的qwen2-1.5b只需要0.9G ，vllm需要4G，不知道是不是量化差异
btw: ollama部署qwen2-1.5b模型是0.9G，显存占用是2G，qwen2-7b模型4G，显存5G，在1070显卡也能跑，但是7b会比较慢
测试环境
- windows11
- python310
- torch2.1
- cuda12.1
- 显卡：RTX-3080-12G
模型来源
https://hf-mirror.com/
测试代码
参考的官方调用
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model ontomodel_path = 'path/to/your/model'
model = AutoModelForCausalLM.from_pretrained(model_path,torch_dtype="auto",device_map="auto"
)
model = model.bfloat16() # 解决量化模型报错
print('model ok')tokenizer = AutoTokenizer.from_pretrained(model_path)
print('tokenizer ok')prompt = "Give me a short introduction to large language model. response using chinese"
messages = [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": prompt}
]
print(prompt)text = tokenizer.apply_chat_template(messages,tokenize=False,add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(device)generated_ids = model.generate(model_inputs.input_ids,max_new_tokens=512
)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)
