Projects

Visual Estimation in VLMs

We implicitly make many rapid estimations of properties like weights and volumes of objects as we navigate the world. Deploying vision-language models (VLMs) in robotics applications requires them to have some of these same abilities. VLMs trained on text-image pairs may fall short on such important tasks because they lack the grounding that comes from real interaction via embodied systems. We create a dataset of precise, hand-measured ground truth values to benchmark a wide variety of models (including robotics foundation models). Our emphasis is to move beyond well-studied spatial estimation abilities. We are also studying the interventions we can do, via tool augmentation or fine-tuning, to improve the current models on these tasks.

Paper

Dataset: coming soon

Low-Contention SM-partitioning for Concurrent Prefill and Decode in Disaggregated Serving

with Dr. Nipun Kwatra, Principal Researcher, Microsoft Research India

Disaggregated serving has become an important LLM serving paradigm due to its ability to meet stringent decode latency targets in large-scale deployments. Although disaggregated serving removes prefill-decode interference by locating them on separate GPUs, the compute utilisation of decode GPUs is poor. Our aim is to reclaim some of the idle compute units on decode GPUs. To this end, we investigate CUDA mechanisms that enable us to partition the compute units and schedule limited prefill workload on a fraction of SMs in decode GPUs with minimum impact on decode latency.

Memory Bandwidth Regulation for LLM Inference of Shared Memory Systems

with Dr. Deepak Gangadharan, Assistant Professor, IIIT Hyderabad

LLMs are increasingly being deployed on affordable, consumer hardware where GPU and CPU share the RAM (integrated GPU) such as embedded device Jetson Nano or compact desktop DGX Spark. LLM inference generates heavy traffic to the memory controller; this can hurt the latency-sensitive tasks running on the CPU. Our aim is to coordinate GPU-CPU memory accesses such that CPU tasks complete within their tight deadlines and GPU gives us good throughput.