ACM on Interactive, Mobile, Wearable, and Ubiquitous Technologies
July 7, 2022
In this paper, we introduce TinyOdom, a framework for training and deploying neural inertial models on URC hardware.
ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
July 7, 2022
earable, network architecture search, neural networks, machine learning, datasets, filters, human activity, head-pose, TinyML, optimization, hardware-in-the-loop
tinyML Summit
March 2022
neural networks, edge computing, IoT platforms, AI-based inference, TinyML, activity detection models
Machine Learning for Health (ML4H)
November 15, 2022
generative model, electrocardiogram, data augmentation
We present CardiacGen, a Deep Learning framework for generating synthetic but physiologically plausible cardiac signals like ECG. Based on the physiology of cardiovascular system function, we propose a modular hierarchical generative model and impose explicit regularizing constraints for training each module using multi-objective loss functions. The model comprises 2 modules, an HRV module focused on producing realistic Heart-Rate-Variability characteristics and a Morphology module focused on generating realistic signal morphologies for different modalities. We empirically show that in addition to having realistic physiological features, the synthetic data from CardiacGen can be used for data augmentation to improve the performance of Deep Learning based classifiers.
We present CardiacGen, a Deep Learning framework for generating synthetic but physiologically plausible cardiac signals like ECG.
ACM Symposium on User
Interface Software and Technology (2023)
October 29, 2023
gesture recognition, finger interaction, vibration sensing, unsupervised adversarial training
Wearable devices like smartwatches and smart wristbands have gained substantial popularity in recent years. However, their small interfaces create inconvenience and limit computing functionality. To fill this gap, we propose ViWatch, which enables robust finger interactions under deployment variations, and relies on a single IMU sensor that is ubiquitous in COTS smartwatches. To this end, we design an unsupervised Siamese adversarial learning method. We built a real-time system on commodity smartwatches and tested it with over one hundred volunteers. Results show that the system accuracy is about 97% over a week. In addition, it is resistant to deployment variations such as different hand shapes, finger activity strengths, and smartwatch positions on the wrist. We also developed a number of mobile applications using our interactive system and conducted a user study where all participants preferred our unsupervised approach to supervised calibration. The demonstration of ViWatch is shown at https://youtu.be/N5-ggvy2qfI.
ACM Transactions on Embedded Computing Systems (2023)
May 31, 2023
neurosymbolic, neural architecture search, TinyML, AutoML, bayesian, platform-aware
Machine learning at the extreme edge has enabled a plethora of intelligent, time-critical, and remote applications. However, deploying interpretable artificial intelligence systems that can perform high-level symbolic reasoning and satisfy the underlying system rules and physics within the tight platform resource constraints is challenging. In this paper, we introduce TinyNS, the first platform-aware neurosymbolic architecture search framework for joint optimization of symbolic and neural operators. TinyNS provides recipes and parsers to automatically write microcontroller code for five types of neurosymbolic models, combining the context awareness and integrity of symbolic techniques with the robustness and performance of machine learning models. TinyNS uses a fast, gradient-free, black-box Bayesian optimizer over discontinuous, conditional, numeric, and categorical search spaces to find the best synergy of symbolic code and neural networks within the hardware resource budget. To guarantee deployability, TinyNS talks to the target hardware during the optimization process. We showcase the utility of TinyNS by deploying microcontroller-class neurosymbolic models through several case studies. In all use cases, TinyNS outperforms purely neural or purely symbolic approaches while guaranteeing execution on real hardware.
We introduce TinyNS, the first platform-aware neurosymbolic architecture search framework for joint optimization of symbolic and neural operators.
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
stressful conversations, stress detection, wearables, physiological sensors, intertial sensors,
December 2020
Stressful conversation is a frequently occurring stressor in our daily life. Stressors not only adversely affect our physical and mental health but also our relationships with family, friends, and coworkers. In this paper, we present a model to automatically detect stressful conversations using wearable physiological and inertial sensors. We conducted a lab and a field study with cohabiting couples to collect ecologically valid sensor data with temporally-precise labels of stressors. We introduce the concept of stress cycles, i.e., the physiological arousal and recovery, within a stress event. We identify several novel features from stress cycles and show that they exhibit distinguishing patterns during stressful conversations when compared to physiological response due to other stressors. We observe that hand gestures also show a distinct pattern when stress occurs due to stressful conversations. We train and test our model using field data collected from 38 participants. Our model can determine whether a detected stress event is due to a stressful conversation with an F1-score of 0.83, using features obtained from only one stress cycle, facilitating intervention delivery within 3.9 minutes since the start of a stressful conversation.
In this paper, we present a model to automatically detect stressful conversations using wearable physiological and intertial sensors.
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
mHealth, brushing detection, flossing detection, hand-to-mouth gestures
June 2021
Ensuring that all the teeth surfaces are adequately covered during daily brushing can reduce the risk of several oral diseases. In this paper, we propose the mTeeth model to detect teeth surfaces being brushed with a manual toothbrush in the natural free-living environment using wrist-worn inertial sensors. To unambiguously label sensor data corresponding to different surfaces and capture all transitions that last only milliseconds, we present a lightweight method to detect the micro-event of brushing strokes that cleanly demarcates transitions among brushing surfaces. Using features extracted from brushing strokes, we propose a Bayesian Ensemble method that leverages the natural hierarchy among teeth surfaces and patterns of transition among them. For training and testing, we enrich a publicly-available wrist-worn inertial sensor dataset collected from the natural environment with time-synchronized precise labels of brushing surface timings and moments of transition. We annotate 10,230 instances of brushing on different surfaces from 114 episodes and evaluate the impact of wide between-person and within-person between-episode variability on machine learning model’s performance for brushing surface detection.
In this paper, we propose the mTeeth model to detect teeth surfaces being brushed with a manual toothbrush in the natural free-living environment using wrist-worn inertial sensors.
IEEE Transactions on Signal Processing
March 3, 2022
Bayesian estimation methods for sparse blind deconvolution problems conventionally employ Bernoulli-Gaussian (BG) prior for modeling sparse sequences and utilize Markov Chain Monte Carlo (MCMC) methods for the estimation of unknowns. However, the discrete nature of the BG model creates computational bottlenecks, preventing efficient exploration of the probability space even with the recently proposed enhanced sampler schemes. To address this issue, we propose an alternative MCMC method by modeling the sparse sequences using the Normal-Inverse-Gamma (NIG) prior. We derive effective Gibbs samplers for this prior and illustrate that the computational burden associated with the BG model can be eliminated by transferring the problem into a completely continuous-valued framework. In addition to sparsity, we also incorporate time and frequency domain constraints on the convolving sequences. We demonstrate the effectiveness of the proposed methods via extensive simulations and characterize computational gains relative to the existing methods that utilize BG modeling.
IEEE Sensors Journal
November 15, 2022
feature projection, Internet of Things, machine learning, microcontrollers, model compression, neural architecture search, neural networks, optimization, sensors, TinyML
This paper highlights the unique requirements of enabling onboard machine learning for microcontroller class devices. We characterize a closed-loopwidely applicable workflow of ML model development for microcontroller-class devices and show that several classes of applications adopt a specific instance of it.
2023 IEEE International Symposium on Antennas and Propagation and USNC-URSI Radio Science Meeting (AP-S/URSI). pp. 53–54.
July 23-28, 2023
antenna measurements, three-dimensional displays, neural networks, transfer functions, radar antennas, nonhomogeneous media, ultra wideband antennas
Ultrawideband (UWB) radar sensors are an emerging biosensing modality that can be used to assess the dielectric properties of internal tissues. Antenna effects, including antenna body interactions limit the sensors ability to isolate the weak returns from the internal tissues. In this paper we develop a data driven calibration method for recovering Green’s function of the multilayered media model of the tissue profiles using an Invertible Neural Network (INN). The proposed INN structure is trained to invert the antenna transfer function to form estimates of the Green’s function modeling returns from internal tissues. We use simulation experiments to assess the effectiveness of the trained INN in antenna transfer function inversion.
In this paper we develop a data driven calibration method for recovering Green’s function of the multilayered media model of the tissue profiles using an Invertible Neural Network (INN).
Sensors (Basel)
MotionSense HRV, accelerometer, mobile health, physical activity, sedentary behavior
February 18, 2021
MotionSense HRV is a wrist-worn accelerometery-based sensor that is paired with a smartphone and is thus capable of measuring the intensity, duration, and frequency of physical activity (PA). However, little information is available on the validity of the MotionSense HRV. Therefore, the purpose of this study was to assess the concurrent validity of the MotionSense HRV in estimating sedentary behavior (SED) and PA. A total of 20 healthy adults (age: 32.5 ± 15.1 years) wore the MotionSense HRV and ActiGraph GT9X accelerometer (GT9X) on their non-dominant wrist for seven consecutive days during free-living conditions. Raw acceleration data from the devices were summarized into average time (min/day) spent in SED and moderate-to-vigorous PA (MVPA). Additionally, using the Cosemed K5 indirect calorimetry system (K5) as a criterion measure, the validity of the MotionSense HRV was examined in simulated free-living conditions. Pearson correlations, mean absolute percent errors (MAPE), Bland-Altman (BA) plots, and equivalence tests were used to examine the validity of the MotionSense HRV against criterion measures. The correlations between the MotionSense HRV and GT9X were high and the MAPE were low for both the SED (r = 0.99, MAPE = 2.4%) and MVPA (r = 0.97, MAPE = 9.1%) estimates under free-living conditions. BA plots illustrated that there was no systematic bias between the MotionSense HRV and criterion measures. The estimates of SED and MVPA from the MotionSense HRV were significantly equivalent to those from the GT9X; the equivalence zones were set at 16.5% for SED and 29% for MVPA. The estimates of SED and PA from the MotionSense HRV were less comparable when compared with those from the K5. The MotionSense HRV yielded comparable estimates for SED and PA when compared with the GT9X accelerometer under free-living conditions. We confirmed the promising application of the MotionSense HRV for monitoring PA patterns for practical and research purposes.
The purpose of this study was to assess the concurrent validity of the MotionSense HRV in estimating sedentary behavior (SED) and PA.
International Conference on Learning Representations (ICLR)
November 13, 2021
privacy, user re-identification, wrist-worn accelerometers
We characterize the re-identification risks inherent in motion sensor data collected from wrist-worn devices in users’ natural environment. We use an open-set formulation, train a deep learning architecture with a new loss function, and apply our model to a new data set.
IEEE/ACM Symposium on Edge Computing (SEC)
December 14, 2021
software security engineering, mobile platform security
Application latency requirements, privacy, and security concerns have naturally pushed computing onto smartphone and IoT devices in a decentralized manner. In response to these demands, researchers have developed micro-runtimes for WebAssembly (Wasm) on IoT devices to enable streaming applications to a runtime that can run the target binaries that are independent of the device. However, the migration of Wasm and the associated security research has neglected the urgent needs of access control on bare-metal, memory management unit (MMU)-less IoT devices that are sensing and actuating upon the physical environment. This paper presents Aerogel, an access control framework that addresses security gaps between the bare-metal IoT devices and the Wasm execution environment concerning access control for sensors, actuators, processor energy usage, and memory usage. In particular, we treat the runtime as a multi-tenant environment, where each Wasm-based application is a tenant. We leverage the inherent sandboxing mechanisms of Wasm to enforce the access control policies to sensors and actuators without trusting the bare-metal operating system. We evaluate our approach on a representative IoT development board: a cortexM4 based development board (nRF52840). Our results show that Aerogel can effectively enforce compute resource and peripheral access control policies while introducing as little as 0.19% to 1.04% runtime overhead and consuming only 18.8% to 45.9% extra energy.
This paper presents Aerogel, an access control framework that addresses security gaps between the bare-metal IoT devices and the Wasm execution environment concerning access control for sensors, actuators, processor energy usage, and memory usage.
Proceedings of the 20th International Conference on Information Processing in Sensor Networks, Pages 386-387
privacy, computing methodologies, machine learning, neural networks
May 18, 2021
The increased availability of on-body sensors gives researchers access to rich time-series data, many of which are related to human health conditions. Sharing such data can allow cross-institutional collaborations that create advanced data-driven models to make inferences on human well-being. However, such data are usually considered privacy-sensitive, and publicly sharing this data may incur significant privacy concerns. In this work, we seek to protect clinical time-series data against membership inference attacks, while maximally retaining the data utility. We achieve this by adding an imperceptible noise to the raw data. Known as adversarial perturbations, the noise is specially trained to force a deep learning model to make inference mistakes (in our case, mispredicting user identities). Our preliminary results show that our solution can better protect the data from membership inference attacks than the baselines, while succeeding in all the designed data quality checks.
In this work, we seek to protect clinical time-series data against membership inference attacks, while maximally retaining the data utility.
ACM on Interactive, Mobile, Wearable, and Ubiquitous Technologies
July 7, 2022
C++
C
Jupyter Notebook
Python
ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
July 7, 2022
C++
C
Python
MATLAB
Jupyter Notebook
https://github.com/ARM-software/mango/tree/main/examples/THIN-Bayes
tinyML Summit
March 2022
Python
Machine Learning for Health (ML4H)
November 15, 2022
Jupyter Notebook
Python
Shell
ACM Transactions on Embedded Computing Systems (2023)
May 31, 2023
C
C++
Assembly
Python
CMake
Makefile
IEEE Transactions on Signal Processing
March 3, 2022
MATLAB
Sensors (Basel)
February 18, 2021
Jupyter Notebook
Python
Shell
International Conference on Learning Representations (ICLR)
November 13, 2021
Jupyter Notebook
The Auritus dataset has 2.43 million inertial samples related to head and full-body movements, consisting of 34 head poses and 9 activities from 45 volunteers. There are three main folders in the dataset, namely Activity Detection, Simple Head Pose and Complex Head Pose. It also contains physiological information about participants in the file Participant_Info.csv, target information for simple and complex head pose in the file HP_target_info.csv, and the IRB approval in the file IRB_Approval.pdf. Simple head pose corresponds to head movements from a origin marker to a target marker and back. Complex head pose corresponds to head movements from a origin marker to target marker A, target marker A to target marker B, and target marker B to origin.
ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
July 7, 2022
TR&D3 will make it possible for temporally-precise mHealth interventions for maintaining health and managing the growing burden of chronic diseases, to be realized on participants’ personal devices with minimal impact on battery life so as not to interfere with routine usage of these devices.
