|
| 1 | +--- |
| 2 | +title: Realtime AI Integrations for Semantic Kernel |
| 3 | +description: Learn about realtime AI integrations available in Semantic Kernel. |
| 4 | +author: eavanvalkenburg |
| 5 | +ms.topic: conceptual |
| 6 | +ms.author: edvan |
| 7 | +ms.date: 02/26/2025 |
| 8 | +ms.service: semantic-kernel |
| 9 | +--- |
| 10 | + |
| 11 | +# Realtime API integrations for Semantic Kernel |
| 12 | + |
| 13 | +The first realtime API integration for Semantic Kernel has been added, it is currently only available in Python and considered experimental. This is because the underlying services are still being developed and are subject to changes. |
| 14 | + |
| 15 | +## Realtime Client abstraction |
| 16 | + |
| 17 | +To support different realtime api's from different vendors, using different protocols, a new client abstraction has been added to the kernel. This client is used to connect to the realtime service and send and receive messages. |
| 18 | +The client is responsible for handling the connection to the service, sending messages, and receiving messages. The client is also responsible for handling any errors that occur during the connection or message sending/receiving process. |
| 19 | + |
| 20 | +### Realtime API |
| 21 | + |
| 22 | +Any realtime client consists of the following methods: |
| 23 | + |
| 24 | +| Method | Description | |
| 25 | +| ---------------- | ------------------------------------------------------------------------------------------------------------------ | |
| 26 | +| `create_session` | Creates a new session | |
| 27 | +| `update_session` | Updates an existing session | |
| 28 | +| `delete_session` | Deletes an existing session | |
| 29 | +| `receive` | This is a asynchronous generator method that listens for messages from the service and yields them as they arrive. | |
| 30 | +| `send` | Sends a message to the service | |
| 31 | + |
| 32 | +## Python implementations |
| 33 | + |
| 34 | +The python version of semantic kernel currently supports the following realtime clients: |
| 35 | + |
| 36 | +| Client | Protocol | Description | |
| 37 | +| ------ | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |
| 38 | +| OpenAI | Websocket | The OpenAI Realtime API is a websocket based api that allows you to send and receive messages in realtime, this connector uses the OpenAI Python package to connect and receive and send messages. | |
| 39 | +| OpenAI | WebRTC | The OpenAI Realtime API is a WebRTC based api that allows you to send and receive messages in realtime, it needs a webRTC compatible audio track at session creation time. | |
| 40 | +| Azure | Websocket | The Azure Realtime API is a websocket based api that allows you to send and receive messages in realtime, this uses the same package as the OpenAI websocket connector. | |
| 41 | + |
| 42 | +## Getting started |
| 43 | + |
| 44 | +To get started with the Realtime API, you need to install the `semantic-kernel` package with the `realtime` extra. |
| 45 | + |
| 46 | +```bash |
| 47 | +pip install semantic-kernel[realtime] |
| 48 | +``` |
| 49 | + |
| 50 | +Then you can create a kernel and add the realtime client to it. |
| 51 | + |
| 52 | +```python |
| 53 | +from semantic_kernel.connectors.ai.open_ai import ( |
| 54 | + AzureRealtimeWebsocket, |
| 55 | + ListenEvents, |
| 56 | + OpenAIRealtimeExecutionSettings, |
| 57 | +) |
| 58 | +from semantic_kernel.contents import RealtimeAudioEvent, RealtimeTextEvent |
| 59 | + |
| 60 | +# this will use environment variables to get the api key, endpoint, api version and deployment name. |
| 61 | +realtime_client = AzureRealtimeWebsocket() |
| 62 | +settings = OpenAIRealtimeExecutionSettings() |
| 63 | +async with realtime_client(settings=settings, create_response=True): |
| 64 | + async for event in realtime_client.receive(): |
| 65 | + match event: |
| 66 | + # receiving a piece of audio |
| 67 | + case RealtimeAudioEvent(): |
| 68 | + await audio_player.add_audio(event.audio) |
| 69 | + # receiving a piece of audio transcript |
| 70 | + case RealtimeTextEvent(): |
| 71 | + # the model returns both audio and transcript of the audio, which we will print |
| 72 | + print(event.text.text, end="") |
| 73 | + case _: |
| 74 | + # OpenAI Specific events |
| 75 | + if event.service_type == ListenEvents.SESSION_UPDATED: |
| 76 | + print("Session updated") |
| 77 | + if event.service_type == ListenEvents.RESPONSE_CREATED: |
| 78 | + print("\nMosscap (transcript): ", end="") |
| 79 | +``` |
| 80 | + |
| 81 | +There are two important things to note, the first is that the `realtime_client` is an async context manager, this means that you can use it in an async function and use `async with` to create the session. |
| 82 | +The second is that the `receive` method is an async generator, this means that you can use it in a for loop to receive messages as they arrive. |
| 83 | + |
| 84 | +In this simple example, we are passing the audio to a unspecified `audio_player` object, and printing the transcript as it arrives. |
| 85 | + |
| 86 | +There is also a `audio_output_callback` parameter on the client creation or on the `receive` method, this callback will be called first, and leads to smoother playback compared to the above example. |
| 87 | + |
| 88 | +See the samples in our repo [link to follow]. |
0 commit comments