So, my new account SomeOddCodeGuy_v2 just got permanently banned; not a shadowban this time, but a proper ban. So, a little backstory: My original reddit account, which had most of my benchmarks and whatnot on it, had gotten Shadowbanned after I posted a link to another reddit post, and
It's been a few weeks since I've posted or made any changes on Wilmer; I haven't stopped or lost interest, but rather I'm about to change jobs and I've been heads down on transition stuff before I leave my current
I was trying to answer someone's question about how Llama.cpp handles offloading with Mixture of Experts models on a regular gaming PC with a 24GB GPU, and ended up spending a few hours in a deep dive.
When I first started Wilmer, it was for a very specific reason: I wanted a semantic router, and one didn't yet exist. The routers that were available were all specifically designed to take the last message, categorize that, and route you that way. I needed more, though; what
So this looks like it could actually be a really fun model https://huggingface.co/microsoft/UserLM-8b I like this little specific purpose LLMs the most because it opens up some neat doors. They likely made this to act as the user-proxy in autogen, and they point out on their
After 3 months, /u/reddit finally messaged me to tell me the account was permanently banned. However, the section that should contain the reason for the ban is empty. It just says Your account has been permanently banned for breaking the rules. This account has been permanently closed. To continue
Every weekend for a while I've put out a release to Wilmer on Sunday; generally a few features I was able to knock out on Saturday and test on Sunday. Almost always using either some combination of local models with Wilmer via Open WebUI, or using Gemini 2.
Someone asked me to run the mxfp4 gguf vs q8, so I figured I'd post the results here too for anyone to see. As expected mxfp4 comes out to a little over half the size of the q8, and the speed is just a bit faster. I expect
For anyone who knows of me, they know that I don't like using coding agents. I have nothing particularly against them, I just don't prefer them. I like the quality and control of workflows, in a direct chat window. You can see as much in my
On thing I've always wanted to do is have Wilmer workflows call themselves, so I can create a form of recursion within the workflows. This allows for a sort of semi-agentic behavior: repeated iterations on a problem with some breakout criteria. Now that may sound like an agent,
A few days ago Stanford dumped a whole pile of AI/ML lectures up on their youtube. They're a pretty good watch if you get bored and want to dive more into this stuff. Stanford OnlineYou can gain access to a world of education through Stanford Online, the
Don't put off unit tests. When I first started building Wilmer, I barely knew any Python, and of course I didn't have Wilmer to help me build it lol. So the early code was nothing shy of a disaster; coming from a C# background, I first
Somehow I've had it this far in life without ever actually using the site. But while I wait for one of my tickets withReddit to finally reach a human so that I can get my account back, outside of discord its one of the few places I can
WilmerAI
So my old Reddit post about my "unorthodox setup" went down with the reddit ship, and figured it was time for an update anyway, so I'm bringing it back. My setup has gotten more complex than I originally planned, built out piecemeal over the past 2.
Ok, for anyone else RDPing into a Windows machine from a Mac that is experiencing a latency between sound and visuals, especially when watching a video: I just went into Settings and set "Graphics Interpolation Level" under "General" to `Medium`, and it had an immediately noticeable
Everyone saying that the new iPhone isn't much, but the fact that they added dedicated MatMul acceleration into the A19 is huge, because this means we'll probably see it in the M5. For folks like me- that's a dream come true. I love my
Links
A quick dump of the benchmarks that I look at and use personally; I've dropped a few that no longer appear to be kept up to date, and grabbed a few newer ones. Code Specific * https://www.swebench.com/ * https://swe-rebench.com/ * https://aider.chat/docs/leaderboards/ Coding
Generative-AI
RAG is really 90% a software development problem, 10% an AI problem. People overcomplicate it on the AI side a lot, but it's a $5 term for a $0.05 concept: give the LLM the answer before it responds to you. At its face, that's simple
Generative-AI
So over the past week I'm suddenly seeing folks posting on LinkedIn about this number that 95% of corps fail to generate significant revenue with AI projects. Honestly, I'd believe it. Personally? I think that a big reason so many AI projects die is because folks
Benchmarks
M3 Ultra Mac Studio 512GB Speeds Qwen3 235b a22b Instruct Q8 in Llama.cpp server (~15k tokens) prompt eval time 4.60 ms per token, 217.29 tokens per second eval time 67.59 ms per token, 14.80 tokens per second total time 146863.82 ms / 15763 tokens (~5k
So back in July, while using a fairly popular commercial VPN, I made a comment on the LocalLlama sub answering someone's question by linking one of my own posts with some benchmarks; something I would do often. After a few minutes, I decided to edit the post to
Found this screenshot from back in 2023; if I remember right, CodeLlama had just come out and I was trying to see how it would do in a coding interview. But look at that old interface for text-gen. The below picture is from the internet, and shows the new interface.
Benchmarks
Mac Model: M3 Ultra Mac Studio 512GB, 80 core GPU First- this model has a shockingly small KV Cache. If any of you saw my post about running Deepseek V3 q4_K_M, you'd have seen that the KV cache buffer in llama.cpp/koboldcpp was 157GB for
Benchmarks
UPDATE 2025-04-13: llama.cpp has had an update that GREATLY improved the prompt processing speed. Please see the new speeds below. Deepseek V3 0324 Q4_K_M w/Flash Attention 4800 token context, responding 552 tokens CtxLimit:4744/8192, Amt:552/4000, Init:0.07s, Process:65.46s (64.02T/
