Skip to main content

Overview

When you need extra audio or video processing between capture and publishing (such as AI noise suppression, beauty filters, or voice changing), the SDK provides a unified processor mechanism. From first principles, a processor is essentially “take one MediaStreamTrack in, output one processed MediaStreamTrack”. The SDK only needs to provide an attachment point on local tracks, while plugins implement the processing logic; the two are decoupled through the TrackProcessor interface. Processors are distributed as separate npm packages, such as the RNN noise suppression plugin @seastart/srtc-plugin-rnnoise.

Attaching and detaching

Local audio/video tracks (LocalMicTrack, LocalCameraTrack, custom tracks, etc.) all provide two methods:
Once attached, it takes effect whether or not the track is published: if not yet published, the processed track is used automatically when publishing; if already published, the SDK switches via replaceTrack without renegotiation, and remote users notice nothing.

Chaining multiple processors

setProcessor accepts an array and chains multiple processors in order (source → P1 → P2 → … → publish). A common combination: noise suppression first, then voice changing.
When you pass an array, the SDK internally uses ProcessorPipeline to combine the processors into one; audio chains share the same AudioContext, reducing the cost of multi-stage processing. You can also use ProcessorPipeline directly:

Writing a custom processor

Implement the TrackProcessor interface to plug in:
ProcessorOptions fields:
When writing an audio processor, reuse options.audioContext if it exists (don’t close it yourself); only close it in destroy if you created the context yourself.

Lifecycle and notes

  • You must startCapture to have a track before calling setProcessor; otherwise it throws.
  • setProcessor is idempotent: calling it again detaches the existing processor before attaching the new one.
  • Automatic reattachment: after switching devices (changeDeviceId) or calling startCapture again, the SDK automatically rebuilds the processor with the new source track, so you don’t need to call setProcessor again.
  • Processors mostly rely on AudioContext/WASM, need HTTPS (or localhost), and may need a user gesture before they can run.