About
Cast AI is an automation platform that operates cloud-native and AI infrastructure at scale. It uses autonomous decision-making in Kubernetes and cloud environments to continuously optimize production performance, reliability, and efficiency. Its Kimchi system automatically matches workloads to cost-efficient, high-performing LLM and serving configurations on customer infrastructure.
Responsibilities
•Increase throughput through continuous batching, speculative decoding, chunked prefill, and kernel-level tuning across vLLM, SGLang, and TensorRT-LLM
•Reduce TTFT and TPOT latency by profiling bottlenecks and addressing compute, memory bandwidth, scheduling, or networking issues
•Improve KV cache utilization through paged attention, prefix caching, eviction policies, cache reuse, and quantized KV
•Quantize weights, activations, and KV using INT8, INT4, and FP8 while measuring quality on real workloads
•Reduce cold starts and memory footprint through faster initialization, smarter weight loading, and tighter memory accounting
•Scale inference across nodes using distributed topologies, network-aware placement, and checkpointing strategies
•Set the technical direction by deciding what to benchmark, adopt, and build, and align the team through writeups and reproducible experiments
Requirements
•5+ years building production ML systems
•A portfolio demonstrating depth in inference or training infrastructure, not just model training notebooks
•Strong Python skills for production services
•Hands-on experience with at least one of vLLM, SGLang, or TensorRT-LLM and an understanding of inference engine performance on different GPUs
•Fluency with quantization tradeoffs, including measuring quality regressions
•Experience with distributed systems, collective communication, sharding strategies, and failure modes in multi-GPU and multi-node setups
•A measurement-focused approach: instrument before optimizing and distinguish real improvements from benchmark artifacts
•Self-direction and comfort with a broad mandate
Conditions
•Competitive salary depending on experience level
•Equity options
•10% of work time for personal projects or self-improvement
•Learning budget for professional and personal development, including international conferences and courses
•Annual hackathon
•Team-building budget and company events
•Equipment budget
•Extra days off
•No visa sponsorship or work permit is provided
•A background check may be conducted at the final recruitment stage through Checkr
