How to (Ab)Use LLM's to manage your KS-LE-B | Part 2 | Shell Access YOLO Edition

NeoonNeoon OG
edited August 27 in Technical

Part 1: https://lowendspirit.com/discussion/10471/how-to-ab-use-your-ks-le-b-for-llm-models/p1

Lets assume you starting fresh.

Grab the dependencies we need.
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev git ccache python3-pip python3.13-venv -y

Grab llama.cpp

cd
git clone https://github.com/ggml-org/llama.cpp.git

Grab huggingface CLI

curl -LsSf https://hf.co/cli/install.sh | bash
export PATH="/root/.local/bin:$PATH"

Build llama.cpp

cmake llama.cpp -B llama.cpp/build
cmake --build llama.cpp/build --config Release --clean-first --target llama-cli llama-server

The build is single threaded, if you got more cores and memory, just add -j 2 for example
While this is building, grab a second SSH session and a model

hf download bloomer010/Ling-3.0-tiny-GGUF --include "*Q4_K_M*" --local-dir models/

When the build finished, copy the binaries

cp llama.cpp/build/bin/llama-* llama.cpp

Start the Server in one SSH Shell
llama.cpp/llama-server --jinja --host 127.0.0.1 --port 8888 --models-dir models/

You can get a simple agent here:
wget https://raw.githubusercontent.com/smol-env/smol/refs/heads/main/smol.py

You have to edit smol.py and change the model to the above one or the one you chosen.
In our case change the model to Ling-3.0-tiny-Q4_K_M

Lets run the agent
python3 smol.py http://127.0.0.1:8888/v1/responses

Example

The agent runs as root and has full shell access, so nothing can go wrong here.
For true lowend servers, host the llama.cpp server somewhere else, the agent itself is lightweight.

This is just an example guide, I have not checked the terminal benchmark on the Ling 3.0 tiny.
There might be better models for this, essentially you can go around and try models as you like.

Free NAT KVM | Free NAT LXC

Comments

Sign In or Register to comment.