Bring your own GPU ยท Part 1
Every token reads the whole model
nvidia-smi showed my 3090 at 100% while my agents got slower. Timing prefill and decode separately showed a card that was mostly waiting on memory.
Topic
Sysadmin
Index
2 entriesSysadmin Aphorisms
Credit for these thoughts goes to Steve Stady and Seth Vidal.