If You Buy A Mac for LLMs, Don't Skimp on RAM

The Macbook M1 Pro 16GB is an alright machine for using smaller AI models, but I'm really feeling the pressure on the RAM side.

This thing can allot 10.7GB of its unified RAM as VRAM, which is fine for the smaller models like gpt4all-j-v1.3-groovy, but 13bs are completely out of the question. For example- ggml-vic13b-q4_0.bin, which is only 7ishGB, won't run; I kept getting a random looking crashed until I realized that's what the out of memory exception looks like. Must be some extra overhead, because I'd think that it would...

I regret not just forking over a little more for the extra RAM when I got this machine. Plus I see folks using models like GPT4-x-Vicuna-30b and get a little jealous =D

I'll probably invest in an NVidia machine for this at some point, but I'm concerned about the power draw. Would prefer not to turn my house into a bonfire on accident.