Full Record of CM5 Running ChatGLM-MNN Big Language Model

[george]

This example is based on the COOLPI CM5 32G memory version, and the system is ubuntu20.04.

• Download ChatGLM-MNN project

git clone https://github.com/wangzhaode/ChatGLM-MNN.git

• Compile MNN library

git clone https://github.com/alibaba/MNN.git
cd MNN
mkdir build && cd build
cmake .. # CPU only
make -j8
cp -r ../include ../../ChatGLM-MNN/
cp libMNN.so ../../ChatGLM-MNN/libs
cp express/libMNN_Express.so ../../ChatGLM-MNN/libs

• Download Models

cd ChatGLM-MNN/resource/models
./download_models.sh int8
cp int8/* ./

• Build

mkdir build
cd build
cmake ..
make -j8

• Run

coolpi@Ubuntu:~/share/ChatGLM-MNN/build$ ./cli_demo    
The device support i8sdot:1, support fp16:1, support i8mm: 0
Can't Find type=2 backend, use 3 instead
load ../resource/tokenizer/slim_vocab.txt ... Done!
load ../resource/models/glm_block_0.mnn model ... Done!
load ../resource/models/glm_block_1.mnn model ... Done!
load ../resource/models/glm_block_2.mnn model ... Done!
load ../resource/models/glm_block_3.mnn model ... Done!
load ../resource/models/glm_block_4.mnn model ... Done!
load ../resource/models/glm_block_5.mnn model ... Done!
load ../resource/models/glm_block_6.mnn model ... Done!
load ../resource/models/glm_block_7.mnn model ... Done!
load ../resource/models/glm_block_8.mnn model ... Done!
load ../resource/models/glm_block_9.mnn model ... Done!
load ../resource/models/glm_block_10.mnn model ... Done!
load ../resource/models/glm_block_11.mnn model ... Done!
load ../resource/models/glm_block_12.mnn model ... Done!
load ../resource/models/glm_block_13.mnn model ... Done!
load ../resource/models/glm_block_14.mnn model ... Done!
load ../resource/models/glm_block_15.mnn model ... Done!
load ../resource/models/glm_block_16.mnn model ... Done!
load ../resource/models/glm_block_17.mnn model ... Done!
load ../resource/models/glm_block_18.mnn model ... Done!
load ../resource/models/glm_block_19.mnn model ... Done!
load ../resource/models/glm_block_20.mnn model ... Done!
load ../resource/models/glm_block_21.mnn model ... Done!
load ../resource/models/glm_block_22.mnn model ... Done!
load ../resource/models/glm_block_23.mnn model ... Done!
load ../resource/models/glm_block_24.mnn model ... Done!
load ../resource/models/glm_block_25.mnn model ... Done!
load ../resource/models/glm_block_26.mnn model ... Done!
load ../resource/models/glm_block_27.mnn model ... Done!
load ../resource/models/lm.mnn model ... Done!

Q: 写一首描绘冬天的诗

A:  漫天雪花飞舞,
寒风呼啸,大地一片沉寂。
冬天的气息弥漫,
银装素裹的世界令人陶醉。

树枝上挂满晶莹剔透的冰霜,
寒风把它们吹得摇曳生姿。
湖面上结满了厚厚的冰,
划着小船的人们欢笑着前行。

冬天是一个美丽而神秘的季节,
让人感受到宁静和祥和。
虽然它带来了一些不便,
比如冷的天气和冰雪的覆盖,
但它也是一年中最美好的时光。<eop>

Q: 

[顾真牛]

@george 目前 MNN 量化不彻底，推理时实际还是f16，所以速度还是慢。
而且 不是很稳定，应该说很不稳定，几条就会奔溃杀死进程。

[顾真牛]

@george 作者是阿里达摩院的MNN框架组的王召德，我有他钉钉，我在天天催更

[george]

@顾真牛 他如果需要环境调试，我们可以提供。