Single Board Computer Hosting
in Help
A long time ago there were a few hosts running single board computers. Does anyone still do this?
With Gemma 3N there may be a use case for a small cluster of SBCs that are always on, and have the facilities (generators) that households don't have at home. It still might be cheaper to use the APIs, but Gemma 3N models are "decent" with the advantage of being designed for on-device use so privacy would be much improved for tasks where slower speeds are okay and the pricing would be a fixed device fee rather than a variable per token fee.
I might just be totally off base with this as well...
Comments
There is a distributed version of llama for clusters. Works, but limited model selection
https://github.com/b4rtaz/distributed-llama
The quality will be in line with size & compute though...aside from trivial chat there will definitely be big quality differences vs api
Microtronix offers micro/mini colocation in Ohio, US: https://clients.microtronix-tech.com/cart.php?a=confproduct&i=0
@jfreak53
Single board vs VPS...
VPS wins in almost every category...
Never make the same mistake twice. There are so many new ones to make.
It’s OK if you disagree with me. I can’t force you to be right.
As one of those hosts who's done whole clusters of SBCs - Don't.
The lifespans are not very good, and often poorly designed. I will admit though some of them do make excellent low cost host nodes, but unless you have experience it's pretty bad. The learning curve to make it work is very steep, and you'll want to know how to make hardware repairs to the boards themselves.
As someone who had setup a 4x Raspberry Pi 3 at home and intended to run them 24/7 as a low cost and low operating cost solution, I agree with above. I had microSD cards dying on and off, regardless of brand and type. When switched to HDD, I had issues with interference with the switching power supply and the USB to Sata controller, which resulted in the drive disconnecting on and off. Switching to SSD caused the loss of a SSD (no idea why).
In the end, decided it's cheaper (specially when you consider my time) to use a full xeon server running proxmox...
Never make the same mistake twice. There are so many new ones to make.
It’s OK if you disagree with me. I can’t force you to be right.
For raspberry PIs I use the Samsung endurance pro as the SD card. I've had cheap SD cards fail before, you should really be using a high endurance ones.
A couple PIs will add up to the same power usage as a modern x86 platform idling
I was using Samsung Evo, Samsung Pro+, Sandisk Ultra micro SD cards. I did not have any mission critical data on the cards so loss of data wasn't a big deal.
Never make the same mistake twice. There are so many new ones to make.
It’s OK if you disagree with me. I can’t force you to be right.
I know the guys over at lagrange.cloud are planning this. Not sure if they released yet. I know he had got some boards in and they were testing a few months ago and trying to figure out chassis situation for them.
Pretty cheap and would be with Ryzen and all that.