Custom tracks

Overview

When the built-in microphone, camera, and screen-sharing capture don’t meet your needs, the Swift SDK provides two extension paths:

Custom audio and video tracks: your app provides the audio or video frames itself
Media processors: process the audio and video frames the SDK has already captured

Common scenarios include:

Publishing video frames generated by an external rendering engine
Sending local PCM audio, BGM, or TTS data into the channel
Beauty filters, virtual background, watermarks, or AI video processing
Voice changing, noise suppression, or enhancement of captured audio

Custom video tracks

Create a custom video track:

let customVideoTrack = srtc.createLocalCustomVideoTrack(desc: "canvas")
try await channel.publishLocalTrack(customVideoTrack)

Then your app keeps calling pushFrame(...) to feed in video frames:

customVideoTrack.pushFrame(pixelBuffer)

The key points:

No startCapture() needed
The SDK doesn’t generate frames for you
You must keep producing CVPixelBuffer values yourself

Suitable for:

Publishing video from Metal / SceneKit / Unity / game engines
Custom composited video
AI-generated frames or the output of image processing

Custom audio tracks

Create a custom audio track:

let customAudioTrack = srtc.createLocalCustomAudioTrack(desc: "bgm")
try await channel.publishLocalTrack(customAudioTrack)

Then keep injecting PCM data:

customAudioTrack.pushAudioBuffer(pcmBuffer)

This kind of track also doesn’t need startCapture(), because the audio source isn’t the SDK’s internal capturer but the AVAudioPCMBuffer you’ve prepared yourself. Suitable for:

Playing background music and publishing it to the channel
Sending PCM data decoded from local files into RTC
Output from speech synthesis, voice cloning, or external DSP engines

Minimal flow summary

Whether it’s custom audio or custom video, the overall model is the same:

let track = srtc.createLocalCustomVideoTrack(desc: "custom")
try await channel.publishLocalTrack(track)

// From here on, your app keeps feeding frames
track.pushFrame(pixelBuffer)

In other words, the SDK handles “publishing and transport”, and you handle “producing the media data”.

Video processor chain

If what you need isn’t “producing new video frames yourself” but “processing existing captured frames”, videoProcessors is the better fit:

final class PassThroughProcessor: VideoProcessor {
    func processVideoFrame(_ frame: VideoFrame) -> VideoFrame? {
        // Here you can replace the pixelBuffer, add filters, or add a watermark
        return frame
    }
}

let cameraTrack = srtc.createLocalCameraTrack(preset: .h720p)
cameraTrack.videoProcessors = [PassThroughProcessor()]
try await cameraTrack.startCapture()
try await channel.publishLocalTrack(cameraTrack)

Processors run serially in array order:

Return a new VideoFrame to pass it on
Return nil to drop the frame

This is better suited for:

Beauty filters
Virtual background
Video watermarks
AI segmentation or visual enhancement

Audio processors

Audio processors are attached to the SRTCEngine main entry point, not to an individual track:

final class MyAudioProcessor: NSObject, AudioProcessor {
    func audioProcessingInitialize(sampleRate sampleRateHz: Int, channels: Int) {}

    func audioProcessingProcess(audioBuffer: AudioBuffer) {
        // You can modify the Float sample values in place
    }

    func audioProcessingRelease() {}
}

srtc.audioCaptureProcessor = MyAudioProcessor()

Note:

audioCaptureProcessor processes audio after capture
audioRenderProcessor processes remote audio before playback
This is a global capability, not a per-track one

So if what you need is “processing only one particular microphone”, first confirm whether this global model matches your expectations.

If you only want to observe remote audio data

Then don’t use AudioProcessor; use AudioRenderer instead:

final class MyTranscriber: AudioRenderer {
    func render(pcmBuffer: AVAudioPCMBuffer) {
        // Send the PCM data off for transcription or recording
    }
}

let renderer = MyTranscriber()
remoteAudioTrack.add(audioRenderer: renderer)

The difference between the two is clear:

AudioProcessor: can modify audio
AudioRenderer: only observes, doesn’t modify

Recommendations

If your goal is to:

Generate audio and video data yourself and publish it: use a custom track
Process local video the SDK has already captured: use videoProcessors
Process captured or playback audio: use audioCaptureProcessor / audioRenderProcessor
Read remote PCM for analysis or transcription: use AudioRenderer

Don’t conflate these mechanisms. They solve four different kinds of problems.

Overview

Web SDK

Android SDK

Swift SDK (iOS/macOS)

C SDK (server/embedded)

Python SDK (server-side AI)

Server API

Overview

Custom video tracks

Custom audio tracks

Minimal flow summary

Video processor chain

Audio processors

If you only want to observe remote audio data

Recommendations

​Overview

​Custom video tracks

​Custom audio tracks

​Minimal flow summary

​Video processor chain

​Audio processors

​If you only want to observe remote audio data

​Recommendations

Overview

Custom video tracks

Custom audio tracks

Minimal flow summary

Video processor chain

Audio processors

If you only want to observe remote audio data

Recommendations