Update SLO/ Predicted latency scheduler for new apis (chat completions, responses, conversations) #2219
Description
What would you like to be added:
Update the SLO/ latency predictor multi plugin for new apis (responses and conversations apis) and also existing apis (chat completions).
Why is this needed:
Support for the openai responses api and conversations api has been added recently (via PR #2133). As part of that PR, some plugin specific updates were also made to the prefix plugin to have correct scheduling when supporting the responses api and conversations api endpoints. However it seems that the SLO/ predicted latency multi plugin only contains logic for the completions api and hence likely needs updates for the newer apis (and potentially even for the chat completions api). Hence opening this issue to track changes to the SLO plugin for all existing and new supported apis (specially chat completions, responses and conversations apis).