SensorBench: Establishing the First Systematic Benchmark for LLM Sensor Processing Capabilities
Sensor data processing is the backbone of modern cyber-physical systems (CPS), underpinning critical applications from healthcare monitoring to industrial analytics. However, the traditional workflow for interpreting and managing sensor data demands profound theoretical knowledge and proficiency in specialized signal-processing tools.
Recent advances in Large Language Models (LLMs) have hinted at a revolutionary potential: using these models as intelligent copilots to develop sensing systems, thereby making advanced sensor data analysis accessible to a broader audience. LLMs have shown promise in analyzing complex health datasets, interpreting motion sensors, and reasoning over spatiotemporal household sensor traces.
To systematically explore this potential, mDOT Center researchers have introduced SensorBench, the first comprehensive benchmark designed to establish a quantifiable objective for evaluating LLMs in coding-based sensor processing.
What is SensorBench?
SensorBench provides a structured evaluation framework for future research by addressing the current challenge of fragmented studies that use varied methodologies and metrics.
The benchmark is meticulous in its design, comprising a diverse range of tasks and sensory types to test various aspects of LLM performance in realistic scenarios. SensorBench incorporates diverse real-world sensor datasets covering modalities such as audio, ECG, PPG, motion, and pressure signals.
Crucially, SensorBench evaluates LLMs not merely on high-level reasoning, but on their ability to perform automated Python coding for digital signal processing (DSP) tasks. LLMs are provided with a Python coding environment and access to established APIs, including numpy and scipy. The tasks are grounded in established resources, such as MATLAB tutorials and DSP textbooks.
Structured Difficulty Levels
To determine the absolute and relative performance of LLMs compared to human experts (Q1), SensorBench categorizes tasks based on difficulty:
1. Single/Compositional: Compositional tasks require integrating multiple processes or signal understanding, while single tasks involve isolated API calls.
2. Parameterized/Non-parameterized: Parameterized tasks involve selecting specific, critical parameters (e.g., setting a stop-band frequency), which heavily determines the processing quality.
Tasks are assigned a difficulty level from 2 to 4, based on the sum of these factors. Evaluation utilizes metrics appropriate for signal processing, such as F1 Score for detection tasks, Signal-to-Distortion Ratio (SDR) for audio reconstruction, and Mean Squared Error (MSE) for signal matching.
The Gap Between Models and Experts
The evaluation of LLMs (including advanced models like GPT-4o, GPT-4, and Llama-3-70b) against human experts revealed significant findings regarding their current capabilities:
• Success in Simple Tasks: Current LLMs demonstrate performance comparable to human experts on simpler tasks, such as Spectral filtering-ECG, Resampling, and Imputation. In these cases, LLMs can often rely on existing world knowledge obtained during training to call established APIs (e.g., using scipy to design a filter).
• Struggle with Complex Tasks: A significant gap exists in harder tasks (difficulties $> 4$) that require iterative problem-solving, multi-step planning, and meticulous parameter tuning. Tasks like Change point detection, Echo cancellation, and Heart rate calculation are particularly challenging for LLMs. Human experts still outperform LLMs by over 60% on these demanding tasks.
• The Parameter Problem: LLM failures often stem from inappropriate parameter selection. For instance, an LLM might incorrectly assume a stop-band frequency in an audio denoising task, a detail that requires careful spectrum analysis and feedback—information that cannot be retrieved solely from training data.
Enhancing Performance Through Self-Verification
Recognizing the need to improve LLM reasoning, SensorBench also investigated whether LLMs could benefit from advanced prompting strategies inspired by the iterative problem-solving mindset of human experts (Q2). Four approaches were tested: Base, Chain-of-Thought (CoT), ReAct, and Self-verification.
The study demonstrated that LLMs benefit significantly from strategies that encourage reflection. Specifically, the adapted self-verification prompting strategy, which asks LLMs to propose tentative solutions, obtain initial feedback (often via an internal sanity checker LLM), and refine their approach, achieved the highest win rates.
Self-verification outperformed all other baselines in 48% of tasks and achieved the lowest failure rate (5.75%) among the prompting methods tested on GPT-4o. This success highlights that guiding the model to reflect on its solutions provides the most gains, particularly for high-difficulty tasks.
In summary, SensorBench provides the community with a crucial framework for evaluating LLM competence in sensory data processing. While self-verification enhances LLMs’ performance relative to baseline strategies, bridging the remaining gap with expert-level performance in compositional and parameterized tasks remains a key area for future research.