Bring your own GPU · Part 3
Room for two
vLLM takes 90% of each GPU before anyone connects. Working out where it goes gave me a limit on requests in flight, and the proxy in front of vLLM now enforces it.
Archive
Posts
Rantings of an old geek.
Index
8 entriesFrom a reading queue to a learning loop
A personal setup that researches one article from my filtered feed and turns it into audio for my commute.
Bring your own GPU · Part 2
Two cards, too many variables
My first benchmark said two RTX 3090s were 21 times slower per token than one. A rerun looked fine, and I had changed too many things to know why. That changed how I test.
Coding in the AI Era
We're coding with AI now. The old concepts assumed the author and the reader are the same person. They're not anymore.
Bring your own GPU · Part 1
Every token reads the whole model
nvidia-smi showed my 3090 at 100% while my agents got slower. Timing prefill and decode separately showed a card that was mostly waiting on memory.
Internet of Minions
A look at AI botnets, insecure devices, DDoS risk, and who needs to fix the mess.
Rise of the machines!
Automation, AI, cloud, and the jobs most exposed to software and machines.
Sysadmin Aphorisms
Credit for these thoughts goes to Steve Stady and Seth Vidal.