Skip to main content

OpenAI API Conformance Report

This page provides a detailed breakdown of Llama Stack's conformance to the OpenAI API specification. The conformance score increases as schema issues are fixed and missing properties are implemented.

Auto-generated

This documentation is auto-generated from the OpenAI API specification comparison.

Summary​

MetricValue
Overall Conformance Score91.7%
Endpoints Implemented29/146
Total Properties Checked3441
Schema/Type Issues184
Missing Properties101
Total Issues to Fix285

Integration Test Coverage​

Overall Test Coverage Score: 43.8%

CategoryCoveredTotalScore
CRUD Operations5771.4%
Conversations5955.6%
Request Parameters212584.0%
Streaming Events165330.2%
Structured Output020.0%
Tools21612.5%

Category Scores​

Categories are sorted by conformance score (lowest first, needing most attention).

CategoryScorePropertiesIssuesMissing
Moderations17.0%53539
Batch39.9%1686041
Completions58.7%46172
Files76.2%4282
Embeddings78.6%1430
Vector stores82.9%3104112
Responses88.9%225250
Models93.3%1510
Chat98.5%40351
Conversations98.9%2165194

Missing Endpoints​

The following OpenAI API endpoints are not yet implemented in Llama Stack:

/assistants​

  • /assistants
  • /assistants/{assistant_id}

/audio​

  • /audio/speech
  • /audio/transcriptions
  • /audio/translations
  • /audio/voice_consents
  • /audio/voice_consents/{consent_id}
  • /audio/voices

/chat​

  • /chat/completions/{completion_id}/messages

/evals​

  • /evals
  • /evals/{eval_id}
  • /evals/{eval_id}/runs
  • /evals/{eval_id}/runs/{run_id}
  • /evals/{eval_id}/runs/{run_id}/output_items
  • /evals/{eval_id}/runs/{run_id}/output_items/{output_item_id}

/fine_tuning​

  • /fine_tuning/alpha/graders/run
  • /fine_tuning/alpha/graders/validate
  • /fine_tuning/checkpoints/{fine_tuned_model_checkpoint}/permissions
  • /fine_tuning/checkpoints/{fine_tuned_model_checkpoint}/permissions/{permission_id}
  • /fine_tuning/jobs
  • /fine_tuning/jobs/{fine_tuning_job_id}
  • /fine_tuning/jobs/{fine_tuning_job_id}/cancel
  • /fine_tuning/jobs/{fine_tuning_job_id}/checkpoints
  • /fine_tuning/jobs/{fine_tuning_job_id}/events
  • /fine_tuning/jobs/{fine_tuning_job_id}/pause
  • /fine_tuning/jobs/{fine_tuning_job_id}/resume

/images​

  • /images/edits
  • /images/generations
  • /images/variations

/organization​

  • /organization/audit_logs
  • /organization/certificates
  • /organization/certificates/activate
  • /organization/certificates/deactivate
  • /organization/certificates/{certificate_id}
  • /organization/costs
  • /organization/groups
  • /organization/groups/{group_id}
  • /organization/groups/{group_id}/roles
  • /organization/groups/{group_id}/roles/{role_id}
  • /organization/groups/{group_id}/users
  • /organization/groups/{group_id}/users/{user_id}
  • /organization/invites
  • /organization/invites/{invite_id}
  • /organization/projects
  • /organization/projects/{project_id}
  • /organization/projects/{project_id}/api_keys
  • /organization/projects/{project_id}/api_keys/{key_id}
  • /organization/projects/{project_id}/archive
  • /organization/projects/{project_id}/certificates
  • /organization/projects/{project_id}/certificates/activate
  • /organization/projects/{project_id}/certificates/deactivate
  • /organization/projects/{project_id}/groups
  • /organization/projects/{project_id}/groups/{group_id}
  • /organization/projects/{project_id}/rate_limits
  • /organization/projects/{project_id}/rate_limits/{rate_limit_id}
  • /organization/projects/{project_id}/service_accounts
  • /organization/projects/{project_id}/service_accounts/{service_account_id}
  • /organization/projects/{project_id}/users
  • /organization/projects/{project_id}/users/{user_id}
  • /organization/roles
  • /organization/roles/{role_id}
  • /organization/usage/audio_speeches
  • /organization/usage/audio_transcriptions
  • /organization/usage/code_interpreter_sessions
  • /organization/usage/completions
  • /organization/usage/embeddings
  • /organization/usage/images
  • /organization/usage/moderations
  • /organization/usage/vector_stores
  • /organization/users
  • /organization/users/{user_id}
  • /organization/users/{user_id}/roles
  • /organization/users/{user_id}/roles/{role_id}

/projects​

  • /projects/{project_id}/groups/{group_id}/roles
  • /projects/{project_id}/groups/{group_id}/roles/{role_id}
  • /projects/{project_id}/roles
  • /projects/{project_id}/roles/{role_id}
  • /projects/{project_id}/users/{user_id}/roles
  • /projects/{project_id}/users/{user_id}/roles/{role_id}

/realtime​

  • /realtime/calls
  • /realtime/calls/{call_id}/accept
  • /realtime/calls/{call_id}/hangup
  • /realtime/calls/{call_id}/refer
  • /realtime/calls/{call_id}/reject
  • /realtime/client_secrets
  • /realtime/sessions
  • /realtime/transcription_sessions

/responses​

  • /responses/input_tokens

/skills​

  • /skills
  • /skills/{skill_id}
  • /skills/{skill_id}/content
  • /skills/{skill_id}/versions
  • /skills/{skill_id}/versions/{version}
  • /skills/{skill_id}/versions/{version}/content

/threads​

  • /threads
  • /threads/runs
  • /threads/{thread_id}
  • /threads/{thread_id}/messages
  • /threads/{thread_id}/messages/{message_id}
  • /threads/{thread_id}/runs
  • /threads/{thread_id}/runs/{run_id}
  • /threads/{thread_id}/runs/{run_id}/cancel
  • /threads/{thread_id}/runs/{run_id}/steps
  • /threads/{thread_id}/runs/{run_id}/steps/{step_id}
  • /threads/{thread_id}/runs/{run_id}/submit_tool_outputs

/uploads​

  • /uploads
  • /uploads/{upload_id}/cancel
  • /uploads/{upload_id}/complete
  • /uploads/{upload_id}/parts

/videos​

  • /videos
  • /videos/characters
  • /videos/characters/{character_id}
  • /videos/edits
  • /videos/extensions
  • /videos/{video_id}
  • /videos/{video_id}/content
  • /videos/{video_id}/remix

Detailed Category Breakdown​

Below is a detailed breakdown of conformance issues and missing properties for each category.

Moderations​

Score: 17.0% · Issues: 5 · Missing: 39

/moderations​

POST

Missing Properties (39)
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.harassment
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.harassment/threatening
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.hate
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.hate/threatening
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.illicit
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.illicit/violent
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.self-harm
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.self-harm/instructions
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.self-harm/intent
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.sexual
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.sexual/minors
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.violence
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.violence/graphic
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.harassment
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.harassment/threatening
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.hate
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.hate/threatening
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.illicit
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.illicit/violent
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.self-harm
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.self-harm/instructions
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.self-harm/intent
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.sexual
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.sexual/minors
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.violence
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.violence/graphic
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.harassment
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.harassment/threatening
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.hate
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.hate/threatening
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.illicit
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.illicit/violent
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.self-harm
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.self-harm/instructions
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.self-harm/intent
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.sexual
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.sexual/minors
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.violence
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.violence/graphic
Schema Issues (5)
PropertyIssues
requestBody.content.application/json.properties.inputUnion variants added: 2
requestBody.content.application/json.properties.modelNullable added (OpenAI non-nullable); Default changed: omni-moderation-latest -> None
responses.200.content.application/json.properties.results.items.properties.categoriesType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.results.items.properties.category_applied_input_typesType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.results.items.properties.category_scoresType removed: ['object']; Union variants added: 2

Batch​

Score: 39.9% · Issues: 60 · Missing: 41

/batches​

GET

Missing Properties (10)
  • responses.200.content.application/json.properties.data.items.properties.errors.properties.data
  • responses.200.content.application/json.properties.data.items.properties.errors.properties.object
  • responses.200.content.application/json.properties.data.items.properties.request_counts.properties.completed
  • responses.200.content.application/json.properties.data.items.properties.request_counts.properties.failed
  • responses.200.content.application/json.properties.data.items.properties.request_counts.properties.total
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.input_tokens
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.input_tokens_details
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.output_tokens
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.output_tokens_details
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.total_tokens
Schema Issues (17)
PropertyIssues
responses.200.content.application/json.properties.data.items.properties.cancelled_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.cancelling_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.completed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.error_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.errorsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.expired_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.failed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.finalizing_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.in_progress_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.modelType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.output_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.request_countsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.usageType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.has_moreDefault changed: None -> False
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2

POST

Missing Properties (11)
  • requestBody.content.application/json.properties.output_expires_after
  • responses.200.content.application/json.properties.errors.properties.data
  • responses.200.content.application/json.properties.errors.properties.object
  • responses.200.content.application/json.properties.request_counts.properties.completed
  • responses.200.content.application/json.properties.request_counts.properties.failed
  • responses.200.content.application/json.properties.request_counts.properties.total
  • responses.200.content.application/json.properties.usage.properties.input_tokens
  • responses.200.content.application/json.properties.usage.properties.input_tokens_details
  • responses.200.content.application/json.properties.usage.properties.output_tokens
  • responses.200.content.application/json.properties.usage.properties.output_tokens_details
  • responses.200.content.application/json.properties.usage.properties.total_tokens
Schema Issues (15)
PropertyIssues
requestBody.content.application/json.properties.endpointEnum removed: ['/v1/responses', '/v1/chat/completions', '/v1/embeddings', '/v1/completions', '/v1/moderations', '/v1/images/generations', '/v1/images/edits', '/v1/videos']
responses.200.content.application/json.properties.cancelled_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.cancelling_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.completed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.error_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.errorsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.expired_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.failed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.finalizing_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.in_progress_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.modelType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.output_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.request_countsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.usageType removed: ['object']; Union variants added: 2

/batches/{batch_id}​

GET

Missing Properties (10)
  • responses.200.content.application/json.properties.errors.properties.data
  • responses.200.content.application/json.properties.errors.properties.object
  • responses.200.content.application/json.properties.request_counts.properties.completed
  • responses.200.content.application/json.properties.request_counts.properties.failed
  • responses.200.content.application/json.properties.request_counts.properties.total
  • responses.200.content.application/json.properties.usage.properties.input_tokens
  • responses.200.content.application/json.properties.usage.properties.input_tokens_details
  • responses.200.content.application/json.properties.usage.properties.output_tokens
  • responses.200.content.application/json.properties.usage.properties.output_tokens_details
  • responses.200.content.application/json.properties.usage.properties.total_tokens
Schema Issues (14)
PropertyIssues
responses.200.content.application/json.properties.cancelled_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.cancelling_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.completed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.error_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.errorsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.expired_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.failed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.finalizing_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.in_progress_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.modelType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.output_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.request_countsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.usageType removed: ['object']; Union variants added: 2

/batches/{batch_id}/cancel​

POST

Missing Properties (10)
  • responses.200.content.application/json.properties.errors.properties.data
  • responses.200.content.application/json.properties.errors.properties.object
  • responses.200.content.application/json.properties.request_counts.properties.completed
  • responses.200.content.application/json.properties.request_counts.properties.failed
  • responses.200.content.application/json.properties.request_counts.properties.total
  • responses.200.content.application/json.properties.usage.properties.input_tokens
  • responses.200.content.application/json.properties.usage.properties.input_tokens_details
  • responses.200.content.application/json.properties.usage.properties.output_tokens
  • responses.200.content.application/json.properties.usage.properties.output_tokens_details
  • responses.200.content.application/json.properties.usage.properties.total_tokens
Schema Issues (14)
PropertyIssues
responses.200.content.application/json.properties.cancelled_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.cancelling_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.completed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.error_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.errorsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.expired_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.failed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.finalizing_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.in_progress_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.modelType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.output_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.request_countsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.usageType removed: ['object']; Union variants added: 2

Completions​

Score: 58.7% · Issues: 17 · Missing: 2

/completions​

POST

Missing Properties (2)
  • responses.200.content.application/json.properties.system_fingerprint
  • responses.200.content.application/json.properties.usage
Schema Issues (17)
PropertyIssues
requestBody.content.application/json.properties.best_ofType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 1 -> None
requestBody.content.application/json.properties.echoType removed: ['boolean']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: False -> None
requestBody.content.application/json.properties.frequency_penaltyType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 0 -> None
requestBody.content.application/json.properties.logit_biasType removed: ['object']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.logprobsType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.max_tokensType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 16 -> None
requestBody.content.application/json.properties.modelType added: ['string']; Union variants removed: 2
requestBody.content.application/json.properties.nType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 1 -> None
requestBody.content.application/json.properties.presence_penaltyType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 0 -> None
requestBody.content.application/json.properties.promptUnion variants added: 4; Default changed: <|endoftext|> -> None
requestBody.content.application/json.properties.seedType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.stopNullable added (OpenAI non-nullable); Union variants added: 3
requestBody.content.application/json.properties.streamType removed: ['boolean']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: False -> None
requestBody.content.application/json.properties.suffixType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.temperatureType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 1 -> None
requestBody.content.application/json.properties.top_pType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 1 -> None
requestBody.content.application/json.properties.userType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2

Files​

Score: 76.2% · Issues: 8 · Missing: 2

/files​

GET

Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.data.items.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.purposeEnum values removed: ['assistants_output', 'batch_output', 'fine-tune', 'fine-tune-results', 'vision', 'user_data']

POST

Missing Properties (2)
  • requestBody.content.multipart/form-data.properties.expires_after.properties.anchor
  • requestBody.content.multipart/form-data.properties.expires_after.properties.seconds
Schema Issues (4)
PropertyIssues
requestBody.content.multipart/form-data.properties.expires_afterType removed: ['object']; Union variants added: 2
requestBody.content.multipart/form-data.properties.purposeEnum values removed: ['fine-tune', 'vision', 'user_data', 'evals']
responses.200.content.application/json.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.purposeEnum values removed: ['assistants_output', 'batch_output', 'fine-tune', 'fine-tune-results', 'vision', 'user_data']

/files/{file_id}​

GET

Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.purposeEnum values removed: ['assistants_output', 'batch_output', 'fine-tune', 'fine-tune-results', 'vision', 'user_data']

Embeddings​

Score: 78.6% · Issues: 3 · Missing: 0

/embeddings​

POST

Schema Issues (3)
PropertyIssues
requestBody.content.application/json.properties.inputUnion variants added: 4
requestBody.content.application/json.properties.modelType added: ['string']; Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.embeddingType removed: ['array']; Union variants added: 2

Vector stores​

Score: 82.9% · Issues: 41 · Missing: 12

/vector_stores​

GET

Missing Properties (2)
  • responses.200.content.application/json.properties.data.items.properties.expires_after.properties.anchor
  • responses.200.content.application/json.properties.data.items.properties.expires_after.properties.days
Schema Issues (3)
PropertyIssues
responses.200.content.application/json.properties.data.items.properties.expires_afterType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.nameDefault changed: None ->
responses.200.content.application/json.properties.data.items.properties.usage_bytesDefault changed: None -> 0

POST

Missing Properties (4)
  • requestBody.content.application/json.properties.expires_after.properties.anchor
  • requestBody.content.application/json.properties.expires_after.properties.days
  • responses.200.content.application/json.properties.expires_after.properties.anchor
  • responses.200.content.application/json.properties.expires_after.properties.days
Schema Issues (8)
PropertyIssues
requestBody.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 2
requestBody.content.application/json.properties.descriptionType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.expires_afterType removed: ['object']; Union variants added: 2
requestBody.content.application/json.properties.file_idsType removed: ['array']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_afterType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.nameDefault changed: None ->
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

/vector_stores/{vector_store_id}​

GET

Missing Properties (2)
  • responses.200.content.application/json.properties.expires_after.properties.anchor
  • responses.200.content.application/json.properties.expires_after.properties.days
Schema Issues (3)
PropertyIssues
responses.200.content.application/json.properties.expires_afterType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.nameDefault changed: None ->
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

POST

Missing Properties (2)
  • responses.200.content.application/json.properties.expires_after.properties.anchor
  • responses.200.content.application/json.properties.expires_after.properties.days
Schema Issues (5)
PropertyIssues
requestBody.content.application/json.properties.expires_afterUnion variants added: 2
requestBody.content.application/json.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_afterType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.nameDefault changed: None ->
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

/vector_stores/{vector_store_id}/file_batches​

POST

Schema Issues (2)
PropertyIssues
requestBody.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 2
requestBody.content.application/json.properties.filesType removed: ['array']; Nullable added (OpenAI non-nullable); Union variants added: 2

/vector_stores/{vector_store_id}/file_batches/{batch_id}/files​

GET

Schema Issues (3)
PropertyIssues
responses.200.content.application/json.properties.data.items.properties.chunking_strategyType removed: ['object']; Union variants added: 3; Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.data.items.properties.usage_bytesDefault changed: None -> 0

/vector_stores/{vector_store_id}/files​

GET

Schema Issues (3)
PropertyIssues
responses.200.content.application/json.properties.data.items.properties.chunking_strategyType removed: ['object']; Union variants added: 3; Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.data.items.properties.usage_bytesDefault changed: None -> 0

POST

Schema Issues (4)
PropertyIssues
requestBody.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 3; Union variants removed: 2
responses.200.content.application/json.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

/vector_stores/{vector_store_id}/files/{file_id}​

GET

Schema Issues (3)
PropertyIssues
responses.200.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 3; Union variants removed: 2
responses.200.content.application/json.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

POST

Schema Issues (4)
PropertyIssues
requestBody.content.application/json.properties.attributesType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 3; Union variants removed: 2
responses.200.content.application/json.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

/vector_stores/{vector_store_id}/search​

POST

Missing Properties (2)
  • requestBody.content.application/json.properties.ranking_options.properties.ranker
  • requestBody.content.application/json.properties.ranking_options.properties.score_threshold
Schema Issues (3)
PropertyIssues
requestBody.content.application/json.properties.filtersUnion variants added: 2
requestBody.content.application/json.properties.queryUnion variants added: 2
requestBody.content.application/json.properties.ranking_optionsType removed: ['object']; Union variants added: 2

Responses​

Score: 88.9% · Issues: 25 · Missing: 0

/responses​

POST

Schema Issues (22)
PropertyIssuesTested
requestBody.content.application/json.properties.inputUnion variants added: 2; Union variants removed: 2Yes
requestBody.content.application/json.properties.modelType added: ['string']; Nullable removed (OpenAI nullable); Union variants removed: 2Yes
requestBody.content.application/json.properties.parallel_tool_callsDefault changed: None -> TrueYes
requestBody.content.application/json.properties.reasoningUnion variants added: 1; Union variants removed: 1Yes
requestBody.content.application/json.properties.service_tierUnion variants added: 2Yes
requestBody.content.application/json.properties.storeDefault changed: None -> TrueYes
requestBody.content.application/json.properties.streamDefault changed: None -> FalseYes
requestBody.content.application/json.properties.stream_optionsUnion variants added: 1; Union variants removed: 1No
requestBody.content.application/json.properties.textUnion variants added: 1; Union variants removed: 1No
requestBody.content.application/json.properties.tool_choiceUnion variants added: 2; Union variants removed: 1Yes
requestBody.content.application/json.properties.truncationUnion variants added: 2Yes
responses.200.content.application/json.properties.errorUnion variants added: 1; Union variants removed: 1Yes
responses.200.content.application/json.properties.incomplete_detailsUnion variants added: 1; Union variants removed: 1Yes
responses.200.content.application/json.properties.metadataUnion variants added: 2Yes
responses.200.content.application/json.properties.output.itemsUnion variants added: 8; Union variants removed: 4Yes
responses.200.content.application/json.properties.parallel_tool_callsDefault changed: None -> TrueYes
responses.200.content.application/json.properties.reasoningUnion variants added: 1; Union variants removed: 1Yes
responses.200.content.application/json.properties.textType added: ['object']No
responses.200.content.application/json.properties.tool_choiceUnion variants added: 3Yes
responses.200.content.application/json.properties.tools.itemsUnion variants added: 4; Union variants removed: 1Yes
responses.200.content.application/json.properties.truncationUnion variants added: 2Yes
responses.200.content.application/json.properties.usageUnion variants added: 1; Union variants removed: 1Yes

/responses/compact​

POST

Schema Issues (3)
PropertyIssuesTested
requestBody.content.application/json.properties.inputUnion variants added: 2; Union variants removed: 1Yes
requestBody.content.application/json.properties.modelType added: ['string']; Union variants removed: 3Yes
responses.200.content.application/json.properties.output.itemsUnion variants added: 4Yes

Models​

Score: 93.3% · Issues: 1 · Missing: 0

/models/{model}​

GET

Schema Issues (1)
PropertyIssues
responses.200.content.application/json.properties.owned_byDefault changed: None -> llama_stack

Chat​

Score: 98.5% · Issues: 5 · Missing: 1

/chat/completions​

GET

Missing Properties (1)
  • parameters.query.metadata
Schema Issues (1)
PropertyIssues
responses.200.content.application/json.properties.data.items.properties.choices.items.properties.message.properties.tool_calls.itemsUnion variants added: 2

POST

Schema Issues (3)
PropertyIssues
responses.200.content.application/json.properties.choices.items.properties.message.properties.tool_calls.itemsUnion variants added: 2
responses.200.content.text/event-stream.properties.choices.items.properties.logprobs.properties.contentType removed: ['array']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.text/event-stream.properties.choices.items.properties.logprobs.properties.refusalType removed: ['array']; Nullable added (OpenAI non-nullable); Union variants added: 2

/chat/completions/{completion_id}​

GET

Schema Issues (1)
PropertyIssues
responses.200.content.application/json.properties.choices.items.properties.message.properties.tool_calls.itemsUnion variants added: 2

Conversations​

Score: 98.9% · Issues: 19 · Missing: 4

/conversations​

POST

Schema Issues (2)
PropertyIssuesTested
requestBody.content.application/json.properties.metadataUnion variants added: 1; Union variants removed: 1Yes
responses.200.content.application/json.properties.metadataUnion variants added: 2Yes

/conversations/{conversation_id}​

DELETE

Schema Issues (2)
PropertyIssuesTested
responses.200.content.application/json.properties.deletedDefault changed: None -> TrueNo
responses.200.content.application/json.properties.objectEnum removed: ['conversation.deleted']No

GET

Schema Issues (1)
PropertyIssuesTested
responses.200.content.application/json.properties.metadataUnion variants added: 2Yes

POST

Schema Issues (2)
PropertyIssuesTested
requestBody.content.application/json.properties.metadataType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2Yes
responses.200.content.application/json.properties.metadataUnion variants added: 2Yes

/conversations/{conversation_id}/items​

GET

Schema Issues (5)
PropertyIssuesTested
responses.200.content.application/json.properties.data.itemsUnion variants added: 10; Union variants removed: 25No
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2No
responses.200.content.application/json.properties.has_moreDefault changed: None -> FalseNo
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2No
responses.200.content.application/json.properties.objectEnum removed: ['list']; Default changed: None -> listNo

POST

Missing Properties (1)
  • parameters.query.include
Schema Issues (6)
PropertyIssuesTested
requestBody.content.application/json.properties.items.itemsUnion variants added: 10; Union variants removed: 3No
responses.200.content.application/json.properties.data.itemsUnion variants added: 10; Union variants removed: 25No
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2No
responses.200.content.application/json.properties.has_moreDefault changed: None -> FalseNo
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2No
responses.200.content.application/json.properties.objectEnum removed: ['list']; Default changed: None -> listNo

/conversations/{conversation_id}/items/{item_id}​

DELETE

Missing Properties (2)
  • responses.200.content.application/json.properties.created_at
  • responses.200.content.application/json.properties.metadata
Schema Issues (1)
PropertyIssuesTested
responses.200.content.application/json.properties.objectEnum removed: ['conversation']; Default changed: conversation -> conversation.item.deletedNo

GET

Missing Properties (1)
  • parameters.query.include

How to Improve Conformance​

To improve conformance scores:

  1. Fix Schema Issues: Update Pydantic models in src/llama_stack_api/ to match OpenAI's schema
  2. Add Missing Properties: Implement missing fields in response models
  3. Add Missing Endpoints: Implement endpoints listed in the Missing Endpoints section

Run the coverage analyzer to check your progress:

python scripts/openai_coverage.py --update

Then regenerate this documentation:

python scripts/generate_openai_coverage_docs.py