Tanmay Gupta

Timeline

1. 2010–2020

   IIT Kanpur and a UIUC PhD

   They study electrical engineering at IIT Kanpur from 2010 to 2014 and become interested in computer vision and machine learning early on. In 2014, they begin a PhD at the University of Illinois Urbana-Champaign under advisor Derek Hoiem, working on joint representation learning for vision and language, and they complete it in 2020.

2. 2017–2018

   Text-to-video generation with CRAFT

   As a research intern at Ai2 in 2017, they work with Aniruddha Kembhavi on one of the earliest deep learning systems for text-to-video generation. The ECCV 2018 paper "Imagine This! Scripts to Compositions to Videos", written with Dustin Schwenk, Ali Farhadi, Derek Hoiem and Kembhavi, presents the CRAFT model and demonstrates it on FLINTSTONES, a video-caption dataset of over 25000 videos.

3. 2020–2024

   Research scientist at Ai2

   They begin their post-PhD career at Ai2 in 2020 as a research scientist on the PRIOR team, working across robotics, multimodal reasoning and vision-language models. They hold the research scientist title through 2024.

4. 2022

   General-purpose vision with GPV-1

   At CVPR 2022, they present GPV-1 with Amita Kamath, Aniruddha Kembhavi and Derek Hoiem. It is a task-agnostic vision-language architecture that learns tasks such as classification, localization, visual question answering and captioning without changes to the architecture or learning process.

5. 2023

   A CVPR 2023 Best Paper

   With Aniruddha Kembhavi, they write Visual Programming: Compositional visual reasoning without training, which introduces VISPROG. The system uses the in-context learning ability of large language models to generate python-like modular programs that call off-the-shelf computer vision models. The paper wins a CVPR 2023 Best Paper award.

6. 2024–2025

   Open Molmo vision-language models

   They are among the 50 authors of Molmo and PixMo, posted to arXiv in September 2024. Molmo is a new family of vision-language models, and PixMo is a collection of new datasets collected without external VLMs. The paper receives a CVPR 2025 Best Paper Honorable Mention.

7. 2025–2026

   Multimodal agents at Ai2

   In 2025, they take the title of senior research scientist at Ai2 and lead multimodal agents research on the PRIOR team. In April 2026, they are first author of MolmoWeb, a family of fully open multimodal web agents that predict the next browser action from a task instruction and a webpage screenshot. They also co-chair the ICCV 2025 Doctoral Consortium with Anna Kukleva.

8. Since 2026

   Computer-use lead at Hark

   In 2026, they join Hark as computer-use lead. They lead a team of engineers and researchers building computer-use agents for Hark's personal intelligence platform, and they work closely with founder Brett Adcock.