AFAIK there isn’t a sparse 35B variant of 3.8, and I would not recommend running a dense 27B model on dual-channel DDR4 unless you are exceptionally patient. 3.6 is good enough for most things anyway, and I wouldn’t expect OP to be doing heavy-duty coding tasks where you would notice the difference.
yellow [she/her]
- 0 Posts
- 3 Comments
- yellow [she/her]@lemmy.blahaj.zonetoSelfhosted@lemmy.world•Any external NPU that can be hooked to a proxmox miniPC?English1·9 days ago
- yellow [she/her]@lemmy.blahaj.zonetoSelfhosted@lemmy.world•Any external NPU that can be hooked to a proxmox miniPC?English3·10 days ago
Any NPU/TPU you can buy is going to be essentially useless for either image diffusion or LLMs. The onboard RAM is both far too small and far too slow (LLM text generation speed relies on RAM speed first and foremost, and both LLMs and image models tend to be, you know, big), and USB isn’t nearly fast enough to help with that, not to mention that software support is pretty much nonexistent. You’d be better off upgrading the GPU to a 3060ti 12gb or something.
P.S. A word of advice, consider using something other than Ollama. Llama.cpp in router mode or llama-swap support pretty much all of the functionality that Ollama does without being crap. Ik_llama.cpp is also nice if you have a CPU/Nvidia setup.
If you wanna make the most out of what you’ve got now, the LFM2.5 series of LLMs are quite good for the small size and fast inference speeds with sizes ranging from 0.2 billion to 8 billion parameters, though their low parameter count means that you’ll probably wanna hook them up to some sort of web search or similar since they won’t have a ton of general knowledge.
If you have at least 32GB of RAM, Qwen3.6 35B is quite a good general-purpose model that runs faster than its parameter count would suggest.
Happy to help! c: