Benchmarks
Previous Post Here are some real-world speeds for the Mac M2 Introduction In my previous post, I showed the raw real-world numbers of what non-cached response times would look like for a Mac Studio M2 Ultra. My goal was to demonstrate how well the machine really handles models at full
One thing I see a lot of when folks are talking about how fast a machine/gpu/whatever is that they will answer the question with this vague "I get 12 tokens per second!", which honestly doesn't feel like it gives a clear answer by itself
Just wanted to toss this out there since I don't see a lot of folks talking about it, but Deepseek 67b Chat has become one of my favorite general purpose models. This is also the first model I've run into that has clearly beaten out, for
IMPORTANT: This is not an exhaustive guide by far. Consider this a quick start tutorial to get you going. For a lot of models, what I have below is all you need. But for some models it looks like there's some wrestling involved with settings, vocab thingies, etc.
I realized I still had gpt4all v2.4.14 on my machine, so I’ve been using it to load up some of the old ggmls that I had laying around, and also see how it does with some of the llama2 models. I’ll miss the ones I can’
Cross-posting this from Reddit So one thing that had really bothered me was that recent Arxiv paper claiming that despite GPT 3 being 175B, and GPT 4 being around 1.7T, somehow 3.5 Turbo was 20b. This had been on my mind for the past couple of days because
So I just got an RTX 4090 and put the machine together. This is going to make a huge difference. Right off the bat I'm jumping into the 13b bandwagon. Vicuna 13b 16k looks awesome, but I've been struggling with the RoPE settings for it. In
I wonder if Llama 2 killed SuperHOTs thanks to NTK scaling. It used to be that the superhots handled that context scaling the best, so I really liked them; especially after seeing someone recommended using NTK with them instead of Linear. Using the superhot version of an older 2048 Llama
So, folks keep asking about running AI models locally, and I figured I'd just put down what I know in one place. Might help someone out there. Short answer: For most machines, I'd try a ggml or gguf of any 7b_q4 to start, just to
So the new gguf format looks cool in that it supposedly automatically calculates the rope-freq for you. However, I do actually have data caps on my internet so I really want to try my hand at converting my existing ggmls. It looks like you can do it a llama.cpp
The Macbook M1 Pro 16GB is an alright machine for using smaller AI models, but I'm really feeling the pressure on the RAM side. This thing can allot 10.7GB of its unified RAM as VRAM, which is fine for the smaller models like gpt4all-j-v1.3-groovy, but 13bs
