SILENCE: Protecting privacy in offloaded speech understanding on resource-constrained devices

This figure shows a bar chart comparing the cost (memory usage and inference time) of using disentanglement-based encoders for privacy-preserving speech understanding on three different platforms: Raspberry Pi 4B (RPI-4B), Jetson TX2 (TX2), and an OnDevice approach (directly on the device). It highlights that disentanglement-based methods require significant resources (648.7MB memory and 12.8s on RPI-4B), making them impractical for resource-constrained devices. The OnDevice approach is included as a baseline, showing the tradeoff between offloading to the cloud and local execution time. The figure supports the paper’s argument that these methods are unsuitable for resource-constrained devices.

This figure illustrates the SILENCE system’s workflow. The user’s speech is processed by the SILENCE system, which utilizes a mask generator to selectively obscure short-term dependencies in the audio signal. This masked audio is then sent to the cloud for intent classification, preserving privacy while maintaining accuracy. The red lines highlight the long-term dependencies crucial for intent recognition, while the green lines show short-term dependencies that are masked to protect privacy.

This figure demonstrates the core idea behind SILENCE, highlighting the asymmetrical dependency between Automatic Speech Recognition (ASR) and Spoken Language Understanding (SLU) tasks. (a) shows that ASR heavily relies on short-term dependencies, focusing on individual phonemes. (b) illustrates that SLU leverages long-term dependencies across the whole utterance to understand intent. (c) presents empirical results showing the SLU accuracy and ASR word error rate under varying levels of masked audio segments, demonstrating that selectively masking portions of the audio doesn’t significantly affect SLU performance but greatly improves privacy protection.

This figure illustrates the two phases of the SILENCE workflow for privacy-preserving speech understanding. The offline phase involves training a mask generator (1a) to selectively obscure sensitive information in the audio and adapting the cloud-based SLU model to handle the masked audio (1b). The online phase uses the trained mask generator to process new audio and sends the masked version to the cloud for intent classification.

This figure illustrates the system architecture of SILENCE and various attack scenarios. It showcases three attack types: passive attacks, active inpainting attacks, and active predicting attacks. Passive attacks utilize pre-trained models like Azure and Whisper to transcribe the masked audio. Active inpainting attacks leverage models like Diffusion or U-Net to reconstruct the masked portions of the audio before transcription. Active predicting attacks use a model like Whisper to predict and fill in the masked sections of the audio, before transcription. The figure highlights how the system’s mask generator is used in each scenario, and it shows the involvement of both normal and malicious users.

This figure compares the performance of different privacy-preserving spoken language understanding (SLU) systems, including SILENCE, VAE, PPSLU, and others. It shows a Pareto-like tradeoff between accuracy (ACC-SLU) and privacy protection (WER-ASR). The results indicate that SILENCE achieves comparable accuracy to other methods while offering significantly better privacy protection, particularly in multi-task settings. It also shows a low entity error rate (EER), further demonstrating that SILENCE is effective at protecting sensitive information within the speech.

This figure shows the effect of the threshold (KL Divergence Lc) on the mask ratio for different mask generator structures (SILENCE-S, SILENCE-M, SILENCE-L). As the threshold increases, indicating a stronger emphasis on privacy, the mask ratio also increases. The figure demonstrates that more complex mask generators (SILENCE-M and SILENCE-L) achieve a higher mask ratio for the same threshold value, suggesting better privacy protection compared to SILENCE-S. The arrow indicates the selected parameters that lead to a good trade-off between privacy and utility.

This figure compares the memory footprint and end-to-end latency of different SLU (spoken language understanding) approaches on two devices, RPI-4B and STM32H7. The approaches compared include: * Local: Running the SLU model entirely on the device. * PPSLU: A state-of-the-art privacy-preserving SLU system that uses disentanglement-based encoders. * SILENCE (S, M, L): Variants of the proposed SILENCE model with varying complexities (S - small, M - medium, L - large). * Random: A baseline approach using a random masking strategy. * Online: Running the model online and uploading audio. The figure clearly demonstrates that SILENCE offers significantly reduced memory footprint and latency compared to other methods, especially PPSLU, while still maintaining reasonable performance. The STM32H7 results show the suitability for resource-constrained devices.

This figure illustrates three different approaches to offloaded speech understanding on resource-constrained devices. (a) shows the basic setup where raw audio is sent to the cloud for processing. (b) shows a previous approach using disentangled encoders to remove sensitive information before sending to the cloud. (c) shows the proposed SILENCE method, which selectively obscures short-term details in the audio signal before sending it to the cloud for processing. The goal is to protect user privacy while maintaining the accuracy of speech understanding.

This figure illustrates the architecture of the SILENCE system. It shows how the system processes an audio input, selectively masking portions of the audio based on the observation that speech understanding (SLU) relies heavily on long-term dependencies in the utterance while privacy-sensitive aspects tend to be localized in shorter timeframes. A mask generator determines which segments of the audio to mask before forwarding it to the cloud for intent extraction. The red line signifies long-term dependencies important for SLU, while the green dotted line depicts shorter-term dependencies that might contain privacy-sensitive data and are selectively masked.

This table presents a comparison of different privacy-preserving techniques for spoken language understanding (SLU) on the Fluent Speech Commands (FSC) dataset. It shows the accuracy of SLU (ACC-SLU) and the word error rate of an automatic speech recognition (ASR) attack (WER-ASR) for several methods: AllOffloaded (no privacy protection), VAE (variational autoencoder), PPSLU (a prior state-of-the-art method), Local (on-device processing), Random (random masking), and SILENCE (the proposed method). The results demonstrate the effectiveness of SILENCE in balancing privacy and utility.

This table presents the performance of different privacy-preserving SLU approaches on a conventional modularized SLU system. It compares the intent classification accuracy (SLU-ACC) achieved by different methods, including the plaintext approach (no privacy preservation), VAE, PPSLU, and three versions of the proposed SILENCE method. The different versions of SILENCE correspond to using the NLU only, a decoupled SLU architecture, and an end-to-end SLU architecture, demonstrating the impact of different levels of integration with the existing SLU model. The results show that even with a modularized approach, SILENCE achieves competitive results.

This table compares the performance of different privacy-preserving SLU approaches across various speech granularities (scenario, action, intent). It shows the accuracy of each approach in correctly identifying the scenario, action, and intent, as well as the word error rate (WER) for SLU and ASR tasks. The ‘/’ indicates that a specific granularity is not supported by that method. Notably, the ‘Local’ approach achieves perfect privacy (WER = 100%) as no data is uploaded to the cloud, but it does not preserve speech understanding performance compared to others. This table highlights the effectiveness of the proposed approach (‘Ours’) in achieving a good balance between privacy and SLU performance across different granularities.