⚡ The main thing in 60 seconds (TL;DR)

⏱ ~ 1 мин чтения
  • While I found that the model does not loop as much as the previous version (0731), I also tried running it on the Q8, but it is very slow (decoding 6-8 current/sec, prefilling 65 current/sec).
🦾 Локальная модель ⚡ 8 ГБ VRAM (RTX 3060 / 4060) 🧠 Контекст: 32K tokens 📦 GGUF

Deepseek V4.1 firmware 1017 on 4rtx3090 + 256ddr4 (Maya engine)

Hardware :

4× NVIDIA RTX 3090, all PCIe x16

256 GB DDR4 memory

Engine: Maya https://github.com/mw00/project-maya
model: orcarouter/DeepSeek-V4.1-Flash-1017-abliterated-GGUF (mix-256G) on 524k context
Pre-fill: 400–600 current/s
Decode: 30–40 current/s (up to 60 current/s maximum) (with dspark)

Hello everyone

I'm still playing with the model, comparing it mainly with the next firmware qwen 3.8 (Q8) and trying to keep myself busy until we get qwen 4. While I found that the model does not loop as much as the previous version (0731), I also tried running it on the Q8, but it is very slow (decoding 6-8 current/sec, prefilling 65 current/sec). I still prefer to use qwen just for the speed I get with the agent task.

Curious to see what speeds others are getting with similar setups with multiple GPUs or a different engine (xolotl hasn't set it up yet, and Strata seems to be using the same engine as Maya for deepseek)?

PS: American diner simulation in Three.js

#SLM #LocalAI #Qwen #LlamaCpp #GPU