Discover sherpa-onnx, a powerful offline speech processing toolkit supporting STT, TTS, speaker ID, and VAD. Cross-platform, 12 languages, privacy-friendly.

Voice technology is rapidly becoming a critical bridge for human-computer interaction, and today we're introducing sherpa-onnx — a powerful and easy-to-use offline speech processing tool. Whether you want to add voice capabilities to embedded devices or develop a cross-platform voice assistant, sherpa-onnx provides a complete solution.

📖 What is sherpa-onnx?

sherpa-onnx is an open-source speech processing toolkit built on the next-generation Kaldi and ONNX Runtime. It supports speech-to-text, text-to-speech, speaker diarization, voice activity detection, and more. Its biggest advantage is that all computation runs locally on the device, requiring no network connection and effectively protecting user privacy.

Developed by the K2-FSA team, the project is committed to providing developers with a unified and efficient speech processing API. It supports up to 12 programming languages and covers deployment scenarios ranging from embedded devices to servers.

🚀 Core Features & Highlights

🔧 Comprehensive Speech Processing

sherpa-onnx is not just a simple speech recognition library — it offers a complete speech processing solution:

  • 🎤 Speech Recognition: Supports both streaming and non-streaming recognition, converting speech to text in real time
  • 🔊 Text-to-Speech: Synthesizes text into natural, fluent speech
  • 📈 Voice Activity Detection: Intelligently detects the start and end points of speech
  • 👥 Speaker Recognition: Distinguishes between different speaker identities
  • ✨ Speech Enhancement & Source Separation: Improves audio quality and separates mixed sources

🌍 Powerful Cross-Platform Support

sherpa-onnx's cross-platform capabilities are impressive: it supports embedded systems, Android, iOS, HarmonyOS, Raspberry Pi, RISC-V, and x86_64 servers. This means you can deploy voice features on virtually any device, from resource-constrained embedded hardware to high-performance servers.

💻 Multi-Language Programming Interfaces

sherpa-onnx supports 12 programming languages, including C++, Python, Java, JavaScript, C#, Kotlin, and Swift. This broad language support makes it easy for developers from any technical background to integrate voice features into their projects.

💡 Practical Application Scenarios

🏠 Smart Home Voice Control

In smart home platforms like Home Assistant, sherpa-onnx can be used to implement offline voice control. Users can control lights, appliances, and other devices through voice commands — without relying on cloud services, resulting in faster response times and better privacy.

🔌 Voice Interaction on Embedded Devices

Thanks to its low resource consumption, sherpa-onnx is well-suited for embedded devices. For example, on a Raspberry Pi, you can easily build an offline voice assistant capable of basic voice interactions.

📱 Voice Features in Mobile Apps

For Android and iOS applications, sherpa-onnx provides dedicated APIs that make it easy to implement speech recognition and synthesis. Developers have shared experiences of quickly building voice interaction demos on Android devices by combining H5 pages with native speech modules.

⚖️ Comparison with Other Speech Tools

Compared with similar tools like PaddleSpeech and Coqui-AI, sherpa-onnx has its unique advantages:

🎯 Deployment Simplicity

sherpa-onnx has fewer dependencies and is easier to deploy. Traditional Kaldi systems require complex dependencies and environment configuration, while sherpa-onnx is built on the ONNX Runtime, greatly simplifying the deployment process.

📊 Resource Consumption

sherpa-onnx has low memory usage and low latency, making it suitable for real-time streaming speech processing. In contrast, some other tools may require more computational resources.

🔒 Offline Capability

Unlike cloud voice services that require internet connectivity, sherpa-onnx runs entirely locally — avoiding network latency and privacy concerns. It's ideal for scenarios with high data security requirements.

🛠️ Usage Tips & Best Practices

📋 Model Selection Recommendations

Choosing the right model for your needs is important: for Chinese recognition, the sherpa-onnx-streaming-zipformer-bilingual-zh-en model is recommended; for English recognition, sherpa-onnx-streaming-paraformer-trilingual-zh-cantonese-en is a solid choice.

⚡ Performance Optimization

On resource-constrained devices, you can tune parameters like audio sampling rate to optimize performance. One developer reported that on a low-spec 4-inch Android screen, dropping the audio sampling rate below 12000 noticeably improved recognition results.

🔑 Custom Keyword Wake-Up

While sherpa-onnx's built-in keyword spotting is highly accurate for predefined keywords, adding new keywords may require fine-tuning. If you need a large number of custom keywords, consider combining it with other solutions, such as iFlytek's customization services.

📥 Installation & Deployment Guide

🔧 Basic Environment Installation

The installation process for sherpa-onnx is relatively straightforward. First, clone the repository:

git clone https://github.com/k2-fsa/sherpa-onnx.git
cd sherpa-onnx

Then install the required dependencies. On Linux/macOS:

sudo apt-get install cmake git # Linux
brew install cmake git # macOS

🐍 Python Environment Installation

If you're using the Python API, you can install via pip:

pip install sherpa-onnx

If you need GPU acceleration (CUDA support):

pip install sherpa-onnx==1.10.16+cuda -f https://k2-fsa.github.io/sherpa/onnx/cuda.html

📦 Model Download

Download the appropriate speech model before use:

# Download the Chinese-English streaming recognition model
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20.tar.bz2
tar xvf sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20.tar.bz2

💻 Basic Usage Example

Here's a simple example of speech recognition using Python:

import sherpa_onnx

# Initialize the recognizer
recognizer = sherpa_onnx.Recognizer()
result = recognizer.recognize("path/to/audio/file.wav")
print(result)

For streaming speech recognition:

# Create a streaming recognition object
stream = recognizer.create_stream()
# Continuously feed audio data
while audio_data := get_audio_chunk():
    stream.accept_waveform(audio_data)

    if recognizer.is_ready(stream):
        recognizer.decode(stream)

        result = recognizer.get_result(stream)
        print(result.text)

🔄 Cross-Platform Compilation

For specific platforms (e.g., macOS), custom compile flags may be needed:

#!/usr/bin/env bash
dir=build-swift-macos
mkdir -p $dir
cd $dir

cmake \
  -DCMAKE_OSX_ARCHITECTURES="x86_64" \
  -DCMAKE_INSTALL_PREFIX=./install \
  -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_SHARED_LIBS=OFF \
  -DSHERPA_ONNX_ENABLE_PYTHON=OFF \
  ../

make -j4
make install

🎉 Conclusion

As a feature-complete, cross-platform offline speech processing tool, sherpa-onnx provides developers with powerful speech technology integration capabilities. Whether for smart home, embedded devices, or mobile applications, you'll find a fitting use case.

With the rise of edge computing and growing privacy awareness, offline speech processing is becoming increasingly important. sherpa-onnx dramatically lowers the barrier to integrating speech technology, enabling more developers to easily add voice interaction features to their products.

If you're looking for a powerful yet flexible speech processing solution, give sherpa-onnx a try — it might just become your project's trusted assistant.

Project URL: 🔗 https://github.com/k2-fsa/sherpa-onnx

Note: This is the English translation of the original Chinese version.