Skip to main content

Provider Compatibility Matrix

{/This file is auto-generated by scripts/provider_compat_matrix.py. Do not edit manually./}

This matrix shows which Responses API features are supported by each inference provider, based on integration test results.

LegendMeaning
✅Tested and passing
❌Tested and failing
⏭️Skipped (unsupported)
—Not tested

Summary​

ProviderTestedPassingFailingCoverage
azure111111085%
bedrock2727021%
ollama2202%
openai1301300100%
vertexai7070054%
vllm3302%
watsonx5353041%

Provider Details​

Models, endpoints, and versions used during test recordings.

ProviderModel(s)EndpointVersion Info
azuregpt-4ollama-stack-test.openai.azure.com, lls-test.openai.azure.com, ogx-test.openai.azure.comopenai sdk: 2.30.0
bedrockopenai.gpt-oss-20b-1:0bedrock-runtime.us-west-2.amazonaws.comopenai sdk: 2.30.0
ollamadeepseek-r1:1.5b—openai sdk: 2.30.0
openaigpt-4o, o4-mini, text-embedding-3-smallapi.openai.comopenai sdk: 2.5.0
vertexaipublishers/google/models/gemini-2.0-flash—openai sdk: 2.5.0, provider: vertexai
vllmQwen/Qwen3-0.6B—openai sdk: 2.5.0, vllm server: 0.18.1rc1.dev197+g0e9358c11
watsonxmeta-llama/llama-3-3-70b-instructus-south.ml.cloud.ibm.comopenai sdk: 2.5.0

Basic Responses​

Featureazurebedrockollamaopenaivertexaivllmwatsonx
extra body guided choice✅✅—✅✅—✅
include logprobs non streaming✅✅—✅✅—✅
include logprobs streaming✅✅—✅✅—✅
include logprobs with function tools✅✅—✅✅—✅
include logprobs with web search✅✅—✅✅—✅
non streaming basic✅✅—✅✅—✅
non streaming image✅⏭️—✅⏭️—⏭️
non streaming multi turn✅✅—✅✅—✅
non streaming multi turn image✅⏭️—✅⏭️—⏭️
streaming basic✅✅—✅✅—✅
streaming incremental content✅✅—✅✅—✅

Compact Responses​

Featureazurebedrockollamaopenaivertexaivllmwatsonx
compact basic conversation✅——✅✅——
compact chain through compaction✅——✅✅——
compact double compaction✅——✅✅——
compact input items hides compaction✅——✅✅——
compact response is retrievable✅——✅✅——
compact roundtrip✅——✅✅——
compact single message✅——✅✅——
compact with previous response id✅——✅✅——
compact with tool calls dropped✅——✅✅——
context management auto compacts large input✅——✅✅——
context management no compact below threshold✅——✅✅——
context management none does not compact✅——✅✅——

Conversation Responses​

Featureazurebedrockollamaopenaivertexaivllmwatsonx
conversation backward compatibility✅✅—✅✅—✅
conversation basic workflow✅✅—✅✅—✅
conversation context loading✅✅—✅✅—✅
conversation error handling⏭️⏭️—✅⏭️—⏭️
conversation multi turn and streaming✅✅—✅✅—✅
Featureazurebedrockollamaopenaivertexaivllmwatsonx
file search filter by category✅——✅⏭️—⏭️
file search filter by date range✅——✅⏭️—⏭️
file search filter by region✅——✅⏭️—⏭️
file search filter compound and✅——✅⏭️—⏭️
file search filter compound or✅——✅⏭️—⏭️
file search streaming events✅——✅⏭️—⏭️
text format✅——✅✅—✅

Mcp Authentication​

Featureazurebedrockollamaopenaivertexaivllmwatsonx
mcp authorization backward compatibility✅✅—✅———
mcp authorization bearer✅✅—✅———

Openai Responses​

Featureazurebedrockollamaopenaivertexaivllmwatsonx
background false is synchronous✅——✅✅—✅
background returns queued✅——✅✅—✅
cancel completed response fails✅——✅⏭️—⏭️
incomplete details length✅——✅⏭️—✅
incomplete details length streaming✅——✅⏭️—✅
incomplete details max iterations exceeded✅——✅✅—✅
incomplete details max iterations exceeded streaming✅——✅✅—✅
incomplete details null when completed✅——✅✅—✅
reasoning effort⏭️——✅⏭️—⏭️
reasoning effort streaming⏭️——✅⏭️—⏭️
streaming includes usage✅——✅✅—✅
streaming invalid base64 image failure code is spec compliant✅——✅⏭️—⏭️
with max output tokens✅——✅✅—⏭️
with parallel tool calls and previous response✅——✅✅—✅
with parallel tool calls disabled✅——✅⏭️—⏭️
with parallel tool calls disabled streaming✅——✅✅—⏭️
with parallel tool calls enabled✅——✅✅—⏭️
with prompt cache key✅——✅✅—✅
with prompt cache key and previous response✅——✅✅—✅
with prompt cache key streaming✅——✅✅—✅
with service tier⏭️——✅⏭️—⏭️
with service tier and previous response⏭️——✅⏭️—⏭️
with service tier auto⏭️——✅⏭️—⏭️
with service tier auto and previous response⏭️——✅⏭️—⏭️
with service tier auto streaming⏭️——✅⏭️—⏭️
with service tier flex⏭️——✅⏭️—⏭️
with service tier flex streaming⏭️——✅⏭️—⏭️
with service tier streaming⏭️——✅⏭️—⏭️
with small max output tokens✅——✅⏭️—⏭️
with stream options and previous response✅——✅✅—✅
with stream options includes usage✅——✅✅—✅
with stream options non streaming✅——✅✅—✅
with top logprobs✅——✅⏭️—✅
with top logprobs and previous response✅——✅⏭️—✅
with top logprobs streaming✅——✅⏭️—✅
with top p✅——✅✅—✅
with top p and previous response✅——✅✅—✅
with top p streaming✅——✅✅—✅
with truncation and previous response✅——✅✅—✅
with truncation disabled✅——✅✅—✅
with truncation disabled streaming✅——✅✅—✅

Prompt Templates​

Featureazurebedrockollamaopenaivertexaivllmwatsonx
basic prompt template✅✅—✅✅—✅
multi variable prompt template✅✅—✅✅—✅
multi version prompt template✅✅—✅✅—✅
prompt template no variables✅✅—✅✅—✅
prompt template with multi turn✅✅—✅✅—✅
prompt template with streaming✅✅—✅✅—✅

Reasoning​

Featureazurebedrockollamaopenaivertexaivllmwatsonx
reasoning basic streaming✅✅⏭️✅✅✅✅
reasoning multi turn passthrough⏭️✅✅✅⏭️✅⏭️
reasoning non streaming⏭️✅✅✅⏭️✅⏭️

Responses Access Control​

Featureazurebedrockollamaopenaivertexaivllmwatsonx
previous response id access control———✅———
user can access own resources after denial———✅———
user cannot access other users response input items———✅———
user cannot delete other users response———✅———
user cannot retrieve other users response———✅———
users have isolated responses———✅———

Responses Errors​

Featureazurebedrockollamaopenaivertexaivllmwatsonx
completed response has no error✅✅—✅✅—✅
invalid base64 image returns image error✅⏭️—✅⏭️—⏭️
invalid image url returns image error✅⏭️—✅⏭️—⏭️
non vision model returns error for image input✅✅—✅⏭️—✅
non vision model with base64 image returns server error✅✅—✅⏭️—✅

Structured Output​

Featureazurebedrockollamaopenaivertexaivllmwatsonx
json schema array of integers✅——✅✅—✅
json schema array of objects✅——✅✅—✅
json schema array of strings✅——✅✅—⏭️
json schema boolean types✅——✅✅—⏭️
json schema float types✅——✅✅—⏭️
json schema integer types✅——✅✅—⏭️
json schema mixed types structures✅——✅✅—✅
json schema nested objects✅——✅✅—✅
json schema string types✅——✅✅—✅

Tool Responses​

Featureazurebedrockollamaopenaivertexaivllmwatsonx
connector resolution mcp tool✅——✅⏭️——
function call ordering 1✅——✅✅——
function call ordering 2✅——✅⏭️——
function call output list file✅——✅✅——
function call output list image✅——✅⏭️——
function call output list text✅——✅✅——
function call output list text multi block✅——✅⏭️——
max tool calls with function tools✅——✅✅——
max tool calls with mcp tools✅——✅⏭️——
mcp tool approval✅——✅⏭️——
multi turn streaming web search✅——✅⏭️——
non streaming custom tool✅——✅✅——
non streaming file search✅——✅⏭️——
non streaming file search empty vector store✅——✅⏭️——
non streaming mcp tool✅——✅⏭️——
non streaming multi turn tool execution✅——✅⏭️——
non streaming web search✅——✅⏭️——
parallel tool calls with function tools✅——✅⏭️——
parallel tool calls with mcp tools✅——✅⏭️——
sequential file search✅——✅⏭️——
sequential mcp tool✅——✅⏭️——
streaming multi turn tool execution✅——✅⏭️——
streaming web search✅——✅⏭️——

Generated by scripts/provider_compat_matrix.py