tl;dr: When running MiniMax M3 Q8_0, dropping temp from 1.0 to 0.8 helped a lot with minor hallucinations and oddities, and disabling MSA also seems fairly promising so far, even though llama.cpp warns that its built-in dense fallback may degrade output. And it might;
After spending the past two weeks redoing all the models around the house, I realized it might make a good topic to chat about. I know that everyone and their brother has their own way to figure out what models they want to run on their hardware, but I figure
Just a quick update- I know it's been quiet on the Wilmer front, but I have a huge update coming soon that I've been working on, and hope to release next weekend. A lot of quality-of-life stuff that I've been using myself
This is one of those "Obvious, but not everyone does it" things that I wanted to call out: the biggest quality of life change that you can make when using an LLM, regardless of whether it's a small locally hosted LLM or a big proprietary LLM,
The reality is that a lot of folks who run open source apis/web front ends on their local LAN tend to run it as plain HTTP; whether its backend llm apis, the front end sites, or whatever other stuff you've tossed in: no TLS anywhere in sight.
I haven't posted or made any updates on Wilmer in like a month, and then I suddenly dropped 3 blog posts all at once. If that doesn't tell you that I don't pay attention to things like trying to game SEO, not sure what
MTP So I decided to test out the new MTP in llama.cpp on Metal using my M2 Ultra, and figured I'd toss the results up here. This isn't meant to show the maximum tps you can get on Mac hardware; I'd have run
If you ever tried to run Llama.cpp on a MacOS device that doesn't have internet on it, you've probably hit the annoying GateKeeper errors that it's downloaded from the internet and you should delete it. Generally I just build from source to avoid
Over the past few days, I've realized that there are a lot of folks out there using LLMs that haven't had an opportunity to dig, even a little, into the basics of how LLMs really work. And I guess that makes sense; for the most part,
Just a random note, but Qwen3.6 35b a3b is putting a smile on my face. This little model feels like a big upgrade over 3.5's 27b or 35b a3b. Also- the Wilmer workflow for OpenCode is really going well. I need to test it more, because
So some year and a half after the request was made for me to put tool calling into Wilmer, I've finally got it in there. First off- it was a huge pain to implement; if I didn't have Wilmer itself and agentic coders to help, I&
In my last post, I mentioned using --image-min-tokens to increase the quality of image responses from Qwen3.5. I went to load Gemma 4 the same way, and hit an error: [58175] srv process_chun: processing image... [58175] encoding image slice... [58175] image slice encoded in 7490 ms
