Fully configure frame processors when they are used directly on an audio stream

[1egoman]

Updates the python sdk so that FrameProcessor-based noise cancellation providers can be used directly on AudioStream, without having to go through the agent's RoomIO to be able to initialize itself with credentials.

For example, with this change, something like the below becomes possible:

stream = rtc.AudioStream.from_track(                                                                                                                   
    track=track,
    sample_rate=SAMPLE_RATE,                                             
    num_channels=CHANNELS,
    noise_cancellation=ai_coustics.audio_enhancement(model=ai_coustics.EnhancerModel.QUAIL_VF_L)  ,
) 

The way this works - Tracks now keep track of which room they are part of (holding a weakref value). When the room a track is in changes, it computes new frame processor options and sends these to any AudioStreams which are associated with the track.

The noise_cancellation_leave_open parameter allows the agents sdk to call this from_track method with a frame processor which remains open across the whole session, and won't be auto-closed when the track is closed.

This goes along with livekit/agents#5867, which removes the relevant event handling logic in the agents sdk. I will follow up with a node version of this once the python one is in a good state.

Todo

• Add some tests for this newly added behavior

[1egoman]

This is the biggest thing still to be determined: I'm not exactly sure what to do when a track is removed from a room while an AudioStream that is within that track has a FrameProcessor registered.

• Initial thought: set empty metadata with something like stream._on_processor_credentials_updated(token="", url=""). However, this causes the ai-coustics plugin to throw an error because the "" string cannot be parsed as a URL.
• Next thought: disable the FrameProcessor by setting enabled to False in this situation (probably also log a warning too that this is being done?). The big problem with this is it would overzelously disable FrameProcessors which don't need credentials to work (like the already-existing Krisp VIVA FrameProcessor).
• Other idea: Maybe modify the FrameProcessor interface to make token / url Optional[str]? But that would be a breaking api change, so that's probably out...

Curious what others think here and if there's an approach which I have missed.

[lukasIO]

I'm curious about this part

Closing the processor doesn't work (the track could get added to another room later)

what prevents us from re-instantiating the processor after it's been closed when/if it gets published to a new room?

Tangentially related (and sorry for opening that can of worms): how does this work if the same track gets published to multiple rooms in parallel ?

[1egoman]

Talked about this in a 1:1, a quick summary - I'm going to add a new _on_credentials_cleared event which will be backwards compatible and will allow this to be signaled so a plugin can clear its active credentials. Re-instantiating a FrameProcessor won't work because a class instance is being passed, not a class itself. I have done this in 07fec79.

For this:

how does this work if the same track gets published to multiple rooms in parallel ?

We discussed it briefly but this already a problem today if a single frame processor were used in two agents sdk instances. The behavior today which this would replicate would be "last one wins"; the credentials used would be the last set which were emitted by any room. I'm not sure if this is necessarily catastrophically bad though for admittedly a fairly significant edge case, and fixing this outright would require a lot more evolution to FrameProcessor. So I think I am going to leave it as is for now.