Support OpenAI `/v1/audio/transcriptions` endpoint for Speech-to-Text (ASR) models

[pierDipi]

What would you like to be added:

Currently, IGW doesn't support OpenAI /v1/audio/transcriptions endpoint for Speech-to-Text (ASR) models, which is commonly used for voice bots.

Refs:

• https://platform.openai.com/docs/api-reference/audio/createTranscription
• https://docs.vllm.ai/en/latest/contributing/model/transcription

In scope:

• Framework and server support

Out of scope:

• individual plugins support

Related to #1980

[rahulgurnani]

/milestone Multimodal Model support

[carmal891]

@pierDipi @rahulgurnani Hi, I'd like to work on implementing the /v1/audio/transcriptions endpoint support. I've analysed the code and would like to propose an approach. Please correct me if wrong and guide me with your feedback

Current architecture flow

client (JSON Request) → Envoy → BBR (parses JSON, sets headers) → EPP (parses JSON) → Director (reads JSON) → Backend (gets JSON)

Current Limitation

The gateway currently assumes all requests are JSON format, but audio transcription requests use multipart/form-data for file uploads. This causes parsing failures in both BBR and EPP components.

Proposed Solution

1: Client Sends Audio Request
A client application sends an HTTP POST request to the /v1/audio/transcriptions endpoint. Audio transcription requests use

1. Multipart/form-data content type
2. Audio file in binary format
3. Model name as a form field
4. Optional parameters like language, prompt, etc.

2: BBR Processes the Request

The BBR intercepts the request and:

1. Detects it's an audio transcription request by checking :

• The request path (/v1/audio/transcriptions)
• The content type (multipart/form-data)

2. Parses the multipart form data to extract the model name from the form fields
3. Sets two HTTP headers for downstream components:
   X-Gateway-Model-Name: for Envoy proxy routing
   x-gateway-model-name-rewrite: for the EPP to use internally
4. Passes through the raw multipart data unchanged to the next stage

3: EPP reads headers
The EPP receives the request with the headers BBR set:

1. EPP HandleRequestHeaders function automatically reads the x-gateway-model-name-rewrite header
2. It stores this model name in the request context as reqCtx.TargetModelName
3. No parsing of the request body happens at this stage

4: EPP handles request body
When the request body arrives, EPP's server logic:

1. Recognises this is an audio request (from earlier detection or checking headers)
2. Skips the normal JSON parsing that would fail on multipart data
3. Stores the raw multipart body in memory for forwarding to the backend
4. Creates an empty JSON structure ({}) to satisfy the Director's expectations

5: Director
The Director component receives the request context and:

1. Uses the pre-populated TargetModelName (set from headers by EPP)
2. Selects which backend ASR server should handle the request

6: Forwarding to ASR Backend
When sending the request to the selected backend server:

1. The system detects this is an audio request (via a flag set earlier)
2. Instead of converting to JSON (existing approach for normal requests), it sends the raw multipart data stored earlier
3. The ASR backend receives the exact multipart format it expects

7: ASR Processing and Response
The ASR backend:

1. Parses the multipart data
2. Extracts and processes the audio file using the specified model
3. Returns the transcription in the requested format (JSON)
4. The response flows back to the client

[pierDipi]

@carmal891

Stores the raw multipart body in memory for forwarding to the backend
Creates an empty JSON structure ({}) to satisfy the Director's expectations

we need to be able to propagate the request details to plugins (even if for now that request type would remain unused), @kfswain @nirrozenbaum do you have a preferred approach on the general direction for removing "JSON assumptions" in the framework?