Cross-posting this from Reddit So one thing that had really bothered me was that recent Arxiv paper claiming that despite GPT 3 being 175B, and GPT 4 being around 1.7T, somehow 3.5 Turbo was 20b. This had been on my mind for the past couple of days because
So I just got an RTX 4090 and put the machine together. This is going to make a huge difference. Right off the bat I'm jumping into the 13b bandwagon. Vicuna 13b 16k looks awesome, but I've been struggling with the RoPE settings for it. In
I wonder if Llama 2 killed SuperHOTs thanks to NTK scaling. It used to be that the superhots handled that context scaling the best, so I really liked them; especially after seeing someone recommended using NTK with them instead of Linear. Using the superhot version of an older 2048 Llama
So, folks keep asking about running AI models locally, and I figured I'd just put down what I know in one place. Might help someone out there. Short answer: For most machines, I'd try a ggml or gguf of any 7b_q4 to start, just to
So the new gguf format looks cool in that it supposedly automatically calculates the rope-freq for you. However, I do actually have data caps on my internet so I really want to try my hand at converting my existing ggmls. It looks like you can do it a llama.cpp
The Macbook M1 Pro 16GB is an alright machine for using smaller AI models, but I'm really feeling the pressure on the RAM side. This thing can allot 10.7GB of its unified RAM as VRAM, which is fine for the smaller models like gpt4all-j-v1.3-groovy, but 13bs
