RF-HOI: Recognize Human-Object Interaction
with Radio Frequency Signals

Lihao Wang, Linlu Gao, Jiacan Yu, Yanyu Lin, Yifan Yin, Jianxin Wang, Tianmin Shu, Renjie Zhao^†

Johns Hopkins University

^†Corresponding author

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 2026

📄 Paper 🌐 ACM DL (coming soon)

RF-HOI recognizes human-object interactions with RF signals in a privacy-preserving way — Using RF signals, RF-HOI understands the user in a privacy-preserving way by recognizing the user's interactions with objects. Unlike human activity recognition (HAR), HOI recognition considers both human actions and the target objects, enabling downstream applications such as goal inference and assistive agents.

Abstract

Recognizing Human-Object Interactions (HOI) is essential for intelligent systems, underpinning applications in virtual and augmented reality, embodied AI, and assistive robotics. However, vision-based HOI methods face challenges in privacy concerns and poor light conditions. In this work, we introduce RF-HOI, the first framework that only uses radio frequency (RF) signals for HOI recognition. A key challenge of RF-HOI is that single-modality RF sensing is insufficient to recognize both actions and the objects being interacted with. RF-HOI addresses this through a novel modality fusion that combines mmWave radar and RFID, enabling simultaneous action recognition and target identification. Another challenge is limited training data across diverse setups, which impairs the generalizability of the recognition model. To overcome this, we develop a simulator that synthesizes multimodal RF data for diverse HOIs at scale, allowing us to fine-tune with only a small amount of real-world data. Experiment results show that RF-HOI outperforms all baselines, approaching vision model performance, and that our diverse synthetic training data can significantly boost our system's performance on real-world scenarios. These results highlight the potential of multimodal RF sensing for robust and privacy-preserving HOI recognition as well as the effectiveness of our RF data synthesis.

Framework Overview

Overview of the RF-HOI framework. RF-HOI uses only RF signals for indoor HOI recognition, leveraging the complementary strengths of mmWave radar (motion sensing) and RFID (object identity). A physics-based simulator synthesizes multimodal RF data at scale for pre-training, and the model is fine-tuned with a small amount of real-world data.

Structure of the modality fusion network. Spatial and temporal features are extracted for each modality, and the two asynchronous modalities are fused at the feature level. The outputs are decoupled into action and object branches to improve generalizability and interpretability.

Data Collection Platform

The RF-HOI platform is built from commercial components: a TI AWR1843AOP mmWave radar, an Impinj R420 RFID reader with four antennas, passive UHF RFID tags, and a Femto Bolt RGB camera (used for the vision baseline and annotation). Data is collected in three environments with increasing multipath richness.

Hardware: mmWave radar, RFID antenna, and camera

Key Results

RF-HOI reaches 96.97% HOI accuracy with full real-world training data, outperforming the no-fusion, RFID-only, and mmWave-only baselines by 19.87%, 36.07%, and 80.06%, respectively.
The average gap to a strong vision (RGB video) baseline is only 2.50%, narrowing to 1.96% with full training data — while avoiding privacy concerns and remaining robust in poor lighting.
Pre-training on our synthetic RF data significantly boosts performance when real-world training samples or setups are limited, and RF-HOI maintains over 90% HOI accuracy under people-induced occlusion.

HOI accuracy compared with single-modality baselines — HOI accuracy vs. single-modality and no-fusion baselines

Comparison with the vision model — RF-HOI approaches the vision model's HOI accuracy

Effect of synthetic pre-training data — Synthetic pre-training boosts accuracy with limited real data

BibTeX

@article{wang2026rfhoi,
  author    = {Wang, Lihao and Gao, Linlu and Yu, Jiacan and Lin, Yanyu and Yin, Yifan and Wang, Jianxin and Shu, Tianmin and Zhao, Renjie},
  title     = {RF-HOI: Recognize Human-Object Interaction with Radio Frequency Signals},
  journal   = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)},
  year      = {2026},
  publisher = {Association for Computing Machinery},
  note      = {To appear}
}

Acknowledgements

We thank the anonymous reviewers for their constructive feedback. We also thank Zhehao Zhang and Jienan Chen for their help with data collection.