• 3 Posts
  • 1.29K Comments
Joined 3 years ago
cake
Cake day: June 4th, 2023

help-circle


  • I have a Strix Halo as well and I have it configured to dynamically allocate everything, so I have run some 100GB+ models as well. The OS itself needs way under a GB without UI so you can get pretty close to the 128GB.

    Currently on Fedora, running llama-swap to start the llama.cpp toolboxes by kyuz0, audio.cpp and ComfyUI.

    Qwen3.8-27b is my favorite model right now for most things, still playing around with Qwen3.8-Flash-Next but not quite there yet.

    Gemma4-31b is also really good with languages and natural writing but I prefer Qwen3.8 for anything programming or logical.
















  • You need to use MTP (Multi token prediction) to get acceptable speeds on a Strix Halo. On llama.cpp you just need to add --spec-type draft-mtp --spec-draft-n-max 3 -np 1 to your arguments.

    You should see around 15-20 t/s in coding tasks at Q8, which will still be much slower than Qwen3-Coder-Next-UD-Q8 since it’s 27b active parameters instead of 3b.

    As for Q8 vs BF16, it usually makes barely to no difference in quality. Speed is higher on Q8 though. Still testing with 3.8 since it just released.