jitllm
An operating system for LLM inference that makes the most of your hardware: generate optimized compute, run models beyond available memory, and orchestrate and
What is jitllm?
An operating system for LLM inference that makes the most of your hardware: generate optimized compute, run models beyond available memory, and orchestrate and migrate layers across CPUs and GPUs at runtime to improve throughput without losing the conversation.