Abstract

In this paper, we address the problem of tactile sim-to-real policy transfer for contact-rich tasks. Existing methods primarily focus on vision-based sensors and emphasize image rendering quality while providing overly simplistic models of force and shear. Consequently, these models exhibit a large sim-to-real gap for many dexterous tasks. Here, we present HydroShear, a non-holonomic hydroelastic tactile simulator that advances the state-of-the-art by modeling: a) stick-slip transitions, b) path-dependent force and shear build up, and c) full SE(3) object-sensor interactions. HydroShear extends hydroelastic contact models using Signed Distance Functions (SDFs) to track the displacements of the on-surface points of an indenter during physical interaction with the sensor membrane. Our approach generates physics-based, computationally efficient force fields from arbitrary watertight geometries while remaining agnostic to the underlying physics engine. In experiments with GelSight Minis, HydroShear more faithfully reproduces real tactile shear compared to existing methods. This fidelity enables zero-shot sim-to-real transfer of reinforcement learning policies across four tasks: peg insertion, bin packing, book shelving for insertion, and drawer pulling for fine gripper control under slip. Our method achieves a 93% average success rate, outperforming policies trained on tactile images (34%) and alternative shear simulation methods (58%-61%).

All videos play at 1x speed

Results Highlights

Peg Insertion

Bin Packing

Book Shelving

Drawer Pulling

Method Overview

Illustration of the teacher-student RL training using Asymmetric Actor-Critic Distillation (AACD). Stage 1: a teacher actor-critic is trained with access to privileged states such as contact forces and object poses. Stage 2: the critic is initialized with the expert critic pretrained from Stage 1 and the student actor that takes high-dimensional inputs (EE pose, relative EE-goal pose, left and right tactile shear) is trained from scratch. The actor-critic is optimized with the PPO RL objective and uses encoder-LSTM-MLP networks. Stage 3: we deploy the student actor in the real world.

Shear Results Highlights

Shear from Different Geometries

Mario Star

Cow (multi-contact patches)

Dumbbell (multi-contact patches)

Ring Torus

Sphere

Sphere (Rolling)

Cross

Translational Shear and Slippage

Front

Side

Roll-induced Shear and Slippage

Front

Side

HydroShear Simulation Overview

Illustration of HydroShear and its pipeline for tactile shear simulation. HydroShear simulates the tactile shear feedback that arises from the physical interaction between the indenter I in (a) and the sensor elastomer E in (b). The goal is to compute the marker displacement fields across the tactile grid query points on the elastomer from (b). HydroShear computes the dilation displacement field in (c) and the shear displacement field in (d) to get the total marker displacement field in (e). The dilation field is computed by identifying the tactile grid points that are in contact with the indenter SDF. Here, the red circle represents the outline of the contact patch. For the shear field, we take the history of indenter poses in the elastomer frame to track the 3D displacementß of the indenter on-surface points, represented by the red arrows in (d), which connects the initial contact location of indenter on-surface points to the current position of the on-surface indenter points while in-penetration to the elastomer SDF.