Sai Haneesh Allu

HRT1: One-Shot Human-to-Robot Trajectory Transfer for Mobile Manipulation

Sai Haneesh Allu*, Jishnu Jaykumar P*, Ninad Khargonkar, Tyler Summers, Jian Yao, Yu Xiang

arXiv preprint · Under review

· Website· PDF· Code·

We introduce a novel system for human-to-robot trajectory transfer that enables robots to manipulate objects by learning from human demonstration videos. The system consists of four modules: a data collection module that collects human demonstration videos from the point of view of a robot using an AR headset; a video understanding module that detects objects and extracts 3D human-hand trajectories; a transfer module that converts a human-hand trajectory into a reference trajectory of a robot end-effector in 3D space; and a trajectory optimization module that solves for a trajectory in the robot configuration space following the transferred end-effector trajectory. Together, these modules enable a robot to watch a human demonstration video once and then repeat the same mobile manipulation task in different environments, even when objects are placed differently from the demonstrations.

@article{2025hrt1, title = {HRT1: One-Shot Human-to-Robot Trajectory Transfer for Mobile Manipulation}, author = {Allu, Sai Haneesh and P, Jishnu Jaykumar and Khargonkar, Ninad and Summers, Tyler and Yao, Jian and Xiang, Yu}, journal = {arXiv}, year = {2025} }

From Local Matches to Global Masks: Template-Guided Instance Detection and Segmentation in Open-World Scenes

Qifan Zhang, Sai Haneesh Allu, Jikai Wang, Yangxiao Lu, Yu Xiang

RSS 2026 · Robotics: Science and Systems

· Website· PDF· Code·

Detecting and segmenting novel object instances in open-world environments is a fundamental problem in robotic perception. Given only a small set of template images, a robot must locate and segment a specific object instance in a cluttered, previously unseen scene. Existing proposal-based approaches are highly sensitive to proposal quality and often fail under occlusion and background clutter. We propose L2G-Det, a local-to-global instance detection framework that bypasses explicit object proposals by leveraging dense patch-level matching between templates and the query image. Locally matched patches generate candidate points, which are refined through a candidate selection module to suppress false positives. The filtered points are then used to prompt an augmented Segment Anything Model (SAM) with instance-specific object tokens, enabling reliable reconstruction of complete instance masks. Experiments demonstrate improved performance over proposal-based methods in challenging open-world settings.

@inproceedings{zhang2026localmatchesglobalmasks, title = {From Local Matches to Global Masks: Novel Instance Detection in Open-World Scenes}, author = {Qifan Zhang and Sai Haneesh Allu and Jikai Wang and Yangxiao Lu and Yu Xiang}, booktitle = {Robotics: Science and Systems (RSS)}, year = {2026} }

A Modular Robotic System for Autonomous Exploration and Semantic Updating

Sai Haneesh Allu, Itay Kadosh, Tyler Summers, Yu Xiang

arXiv preprint · Under submission

· Website· PDF· Code·

We present a modular robotic system for autonomous exploration and semantic updating of large-scale unknown environments. Our approach enables a mobile robot to build, revisit, and update a hybrid semantic map that integrates a 2D occupancy grid for geometry with a topological graph for object semantics. Unlike prior methods that rely on manual teleoperation or precollected datasets, our two-phase approach achieves end-to-end autonomy: first, a modified frontier-based exploration algorithm with dynamic search windows constructs a geometric map; second, using a greedy trajectory planner, environments are revisited and object semantics are updated using open-vocabulary object detection and segmentation. This modular system, compatible with any metric SLAM framework, supports continuous operation by efficiently updating the semantic graph to reflect short-term and long-term changes such as object relocation, removal, or addition. We validate the approach on a Fetch robot in real-world indoor environments of approximately 8,500m² and 117m², demonstrating robust and scalable semantic mapping and continuous adaptation.

@article{allu2024modular, title = {A Modular Robotic System for Autonomous Exploration and Semantic Updating}, author = {Allu, Sai Haneesh and Kadosh, Itay and Summers, Tyler and Xiang, Yu}, year = {2026} }

Grasping Trajectory Optimization with Point Clouds

Yu Xiang, Sai Haneesh Allu, Rohith Peddi, Tyler Summers, Vibhav Gogate

IROS 2024Oral

· Website· PDF· Code·

We introduce a new trajectory optimization method for robotic grasping based on a point-cloud representation of robots and task spaces. Robots are represented by 3D points on their link surfaces, and the task space is represented by a point cloud obtained from depth sensors. Using this representation, goal reaching in grasping can be formulated as point matching, while collision avoidance is efficiently achieved by querying the signed distance values of the robot points in the signed distance field of the scene points. Consequently, a constrained nonlinear optimization problem is formulated to solve the joint motion and grasp planning problem. The advantage of our method is that the point-cloud representation is general enough to be used with any robot in any environment. We demonstrate the effectiveness of our method on a tabletop scene and a shelf scene for grasping with a Fetch mobile manipulator and a Franka Panda arm.

@inproceedings{xiang2024grasping, title = {Grasping Trajectory Optimization with Point Clouds}, author = {Xiang, Yu and Allu, Sai Haneesh and Peddi, Rohith and Summers, Tyler and Gogate, Vibhav}, booktitle = {IROS}, year = {2024} }

SceneReplica: Benchmarking Real-World Robot Manipulation

Ninad Khargonkar*, Sai Haneesh Allu*, Yangxiao Lu, Jishnu Jaykumar P, Balakrishnan Prabhakaran, Yu Xiang

ICRA 2024Oral

· Website· PDF· Code·

We present a new reproducible benchmark for evaluating robot manipulation in the real world, specifically focusing on a pick-and-place task. Our benchmark uses the YCB object set, a commonly used dataset in the robotics community, to ensure that our results are comparable to other studies. The benchmark is designed to be easily reproducible in the real world, making it accessible to researchers and practitioners. We also provide experimental results and analyses for model-based and model-free 6D robotic grasping, where representative algorithms are evaluated for object perception, grasp planning, and motion planning. By providing a standardized evaluation framework, researchers can more easily compare different techniques and algorithms, leading to faster progress in developing robot manipulation methods.

@inproceedings{khargonkar2024scenereplica, title = {SceneReplica: Benchmarking Real-World Robot Manipulation}, author = {Khargonkar, Ninad and Allu, Sai Haneesh and Lu, Yangxiao and P, Jishnu Jaykumar and Prabhakaran, Balakrishnan and Xiang, Yu}, booktitle = {ICRA}, year = {2024} }

Formation Control of Quadcopters

Sai Haneesh Allu

M.S. Thesis, IIT Delhi, 2020

· Thesis· Video· Code

This study investigates various formation control algorithms and implements them on an experimental platform, with the ultimate goal of target interception by choosing the best-suited algorithm. The open-source nanoquadcopter platform Crazyflie 2.0 was chosen for experimentation, and the ArduPilot flight stack along with DroneKit software-in-the-loop were used for simulation. The work studies virtual structure, leader-follower, and graph-theoretic methods of formation control, designs controllers for each, and compares their performance in formation maintenance. The comparison shows that the graph-theoretic method is best suited for formation maintenance, and target interception is simulated using this method. Velocity- and trajectory-based formation control via optimization techniques are proposed as future work.

Sai Haneesh Allu

Robots that work outside the lab.

News

Publications

Industry Experience

Service & Teaching