So I've been digging into agents a bit. Specifically, been looking into AutoGen and CrewAI. These are super cool. I'm not sure what I'd actually use them for, but between the two I think I like CrewAI the best between the two. Next I&
For the past few months I've been working on a quiet little project on the weekends, whenever I can scrounge up time, and part of that project involves looking for the best models of each domain. Of course, there are some great coding, medical, math, etc finetunes, but
So after seeing a lot of folks recommending SillyTavern as a good front end for APIs, I finally decided to give it a better try. I mostly have been using Oobabooga, and while I had ST installed from many months ago, I never put a lot of time into understanding
Benchmarks
Previous Post Here are some real-world speeds for the Mac M2 Introduction In my previous post, I showed the raw real-world numbers of what non-cached response times would look like for a Mac Studio M2 Ultra. My goal was to demonstrate how well the machine really handles
One thing I see a lot of when folks are talking about how fast a machine/gpu/whatever is that they will answer the question with this vague "I get 12 tokens per second!", which honestly doesn't feel like it gives a clear answer by itself
Just wanted to toss this out there since I don't see a lot of folks talking about it, but Deepseek 67b Chat has become one of my favorite general purpose models. This is also the first model I've run into that has clearly beaten out, for
IMPORTANT: This is not an exhaustive guide by far. Consider this a quick start tutorial to get you going. For a lot of models, what I have below is all you need. But for some models it looks like there's some wrestling involved with settings, vocab thingies, etc.
I realized I still had gpt4all v2.4.14 on my machine, so I’ve been using it to load up some of the old ggmls that I had laying around, and also see how it does with some of the llama2 models. I’ll miss the ones I can’
Cross-posting this from Reddit So one thing that had really bothered me was that recent Arxiv paper claiming that despite GPT 3 being 175B, and GPT 4 being around 1.7T, somehow 3.5 Turbo was 20b. This had been on my mind for the past couple of days
So I just got an RTX 4090 and put the machine together. This is going to make a huge difference. Right off the bat I'm jumping into the 13b bandwagon. Vicuna 13b 16k looks awesome, but I've been struggling with the RoPE settings for it. In
I wonder if Llama 2 killed SuperHOTs thanks to NTK scaling. It used to be that the superhots handled that context scaling the best, so I really liked them; especially after seeing someone recommended using NTK with them instead of Linear. Using the superhot version of an older 2048 Llama
So, folks keep asking about running AI models locally, and I figured I'd just put down what I know in one place. Might help someone out there. Short answer: For most machines, I'd try a ggml or gguf of any 7b_q4 to start, just to
