Skip to main content

OpenAI API Conformance Report

This page provides a detailed breakdown of Llama Stack's conformance to the OpenAI API specification. The conformance score increases as schema issues are fixed and missing properties are implemented.

Auto-generated

This documentation is auto-generated from the OpenAI API specification comparison.

Summary​

MetricValue
Overall Conformance Score87.9%
Endpoints Implemented28/146
Total Properties Checked3441
Schema/Type Issues288
Missing Properties130
Total Issues to Fix418

Integration Test Coverage​

Overall Test Coverage Score: 44.1%

CategoryCoveredTotalScore
CRUD Operations5683.3%
Conversations5955.6%
Request Parameters212584.0%
Streaming Events165330.2%
Structured Output020.0%
Tools21612.5%

Category Scores​

Categories are sorted by conformance score (lowest first, needing most attention).

CategoryScorePropertiesIssuesMissing
Moderations15.1%53639
Batch39.3%1686141
Completions56.5%46182
Vector stores61.3%31010614
Embeddings64.3%1450
Files66.7%4286
Models66.7%1505
Chat87.1%4033319
Responses87.1%225290
Conversations98.8%2165224

Missing Endpoints​

The following OpenAI API endpoints are not yet implemented in Llama Stack:

/assistants​

  • /assistants
  • /assistants/{assistant_id}

/audio​

  • /audio/speech
  • /audio/transcriptions
  • /audio/translations
  • /audio/voice_consents
  • /audio/voice_consents/{consent_id}
  • /audio/voices

/chat​

  • /chat/completions/{completion_id}/messages

/evals​

  • /evals
  • /evals/{eval_id}
  • /evals/{eval_id}/runs
  • /evals/{eval_id}/runs/{run_id}
  • /evals/{eval_id}/runs/{run_id}/output_items
  • /evals/{eval_id}/runs/{run_id}/output_items/{output_item_id}

/fine_tuning​

  • /fine_tuning/alpha/graders/run
  • /fine_tuning/alpha/graders/validate
  • /fine_tuning/checkpoints/{fine_tuned_model_checkpoint}/permissions
  • /fine_tuning/checkpoints/{fine_tuned_model_checkpoint}/permissions/{permission_id}
  • /fine_tuning/jobs
  • /fine_tuning/jobs/{fine_tuning_job_id}
  • /fine_tuning/jobs/{fine_tuning_job_id}/cancel
  • /fine_tuning/jobs/{fine_tuning_job_id}/checkpoints
  • /fine_tuning/jobs/{fine_tuning_job_id}/events
  • /fine_tuning/jobs/{fine_tuning_job_id}/pause
  • /fine_tuning/jobs/{fine_tuning_job_id}/resume

/images​

  • /images/edits
  • /images/generations
  • /images/variations

/organization​

  • /organization/audit_logs
  • /organization/certificates
  • /organization/certificates/activate
  • /organization/certificates/deactivate
  • /organization/certificates/{certificate_id}
  • /organization/costs
  • /organization/groups
  • /organization/groups/{group_id}
  • /organization/groups/{group_id}/roles
  • /organization/groups/{group_id}/roles/{role_id}
  • /organization/groups/{group_id}/users
  • /organization/groups/{group_id}/users/{user_id}
  • /organization/invites
  • /organization/invites/{invite_id}
  • /organization/projects
  • /organization/projects/{project_id}
  • /organization/projects/{project_id}/api_keys
  • /organization/projects/{project_id}/api_keys/{key_id}
  • /organization/projects/{project_id}/archive
  • /organization/projects/{project_id}/certificates
  • /organization/projects/{project_id}/certificates/activate
  • /organization/projects/{project_id}/certificates/deactivate
  • /organization/projects/{project_id}/groups
  • /organization/projects/{project_id}/groups/{group_id}
  • /organization/projects/{project_id}/rate_limits
  • /organization/projects/{project_id}/rate_limits/{rate_limit_id}
  • /organization/projects/{project_id}/service_accounts
  • /organization/projects/{project_id}/service_accounts/{service_account_id}
  • /organization/projects/{project_id}/users
  • /organization/projects/{project_id}/users/{user_id}
  • /organization/roles
  • /organization/roles/{role_id}
  • /organization/usage/audio_speeches
  • /organization/usage/audio_transcriptions
  • /organization/usage/code_interpreter_sessions
  • /organization/usage/completions
  • /organization/usage/embeddings
  • /organization/usage/images
  • /organization/usage/moderations
  • /organization/usage/vector_stores
  • /organization/users
  • /organization/users/{user_id}
  • /organization/users/{user_id}/roles
  • /organization/users/{user_id}/roles/{role_id}

/projects​

  • /projects/{project_id}/groups/{group_id}/roles
  • /projects/{project_id}/groups/{group_id}/roles/{role_id}
  • /projects/{project_id}/roles
  • /projects/{project_id}/roles/{role_id}
  • /projects/{project_id}/users/{user_id}/roles
  • /projects/{project_id}/users/{user_id}/roles/{role_id}

/realtime​

  • /realtime/calls
  • /realtime/calls/{call_id}/accept
  • /realtime/calls/{call_id}/hangup
  • /realtime/calls/{call_id}/refer
  • /realtime/calls/{call_id}/reject
  • /realtime/client_secrets
  • /realtime/sessions
  • /realtime/transcription_sessions

/responses​

  • /responses/compact
  • /responses/input_tokens

/skills​

  • /skills
  • /skills/{skill_id}
  • /skills/{skill_id}/content
  • /skills/{skill_id}/versions
  • /skills/{skill_id}/versions/{version}
  • /skills/{skill_id}/versions/{version}/content

/threads​

  • /threads
  • /threads/runs
  • /threads/{thread_id}
  • /threads/{thread_id}/messages
  • /threads/{thread_id}/messages/{message_id}
  • /threads/{thread_id}/runs
  • /threads/{thread_id}/runs/{run_id}
  • /threads/{thread_id}/runs/{run_id}/cancel
  • /threads/{thread_id}/runs/{run_id}/steps
  • /threads/{thread_id}/runs/{run_id}/steps/{step_id}
  • /threads/{thread_id}/runs/{run_id}/submit_tool_outputs

/uploads​

  • /uploads
  • /uploads/{upload_id}/cancel
  • /uploads/{upload_id}/complete
  • /uploads/{upload_id}/parts

/videos​

  • /videos
  • /videos/characters
  • /videos/characters/{character_id}
  • /videos/edits
  • /videos/extensions
  • /videos/{video_id}
  • /videos/{video_id}/content
  • /videos/{video_id}/remix

Detailed Category Breakdown​

Below is a detailed breakdown of conformance issues and missing properties for each category.

Moderations​

Score: 15.1% · Issues: 6 · Missing: 39

/moderations​

POST

Missing Properties (39)
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.harassment
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.harassment/threatening
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.hate
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.hate/threatening
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.illicit
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.illicit/violent
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.self-harm
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.self-harm/instructions
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.self-harm/intent
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.sexual
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.sexual/minors
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.violence
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.violence/graphic
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.harassment
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.harassment/threatening
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.hate
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.hate/threatening
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.illicit
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.illicit/violent
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.self-harm
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.self-harm/instructions
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.self-harm/intent
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.sexual
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.sexual/minors
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.violence
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.violence/graphic
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.harassment
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.harassment/threatening
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.hate
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.hate/threatening
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.illicit
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.illicit/violent
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.self-harm
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.self-harm/instructions
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.self-harm/intent
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.sexual
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.sexual/minors
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.violence
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.violence/graphic
Schema Issues (6)
PropertyIssues
requestBody.content.application/json.properties.inputUnion variants added: 2
requestBody.content.application/json.properties.modelNullable added (OpenAI non-nullable); Default changed: omni-moderation-latest -> None
responses.200.content.application/json.properties.results.itemsType removed: ['object']
responses.200.content.application/json.properties.results.items.properties.categoriesType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.results.items.properties.category_applied_input_typesType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.results.items.properties.category_scoresType removed: ['object']; Union variants added: 2

Batch​

Score: 39.3% · Issues: 61 · Missing: 41

/batches​

GET

Missing Properties (10)
  • responses.200.content.application/json.properties.data.items.properties.errors.properties.data
  • responses.200.content.application/json.properties.data.items.properties.errors.properties.object
  • responses.200.content.application/json.properties.data.items.properties.request_counts.properties.completed
  • responses.200.content.application/json.properties.data.items.properties.request_counts.properties.failed
  • responses.200.content.application/json.properties.data.items.properties.request_counts.properties.total
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.input_tokens
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.input_tokens_details
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.output_tokens
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.output_tokens_details
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.total_tokens
Schema Issues (18)
PropertyIssues
responses.200.content.application/json.properties.data.itemsType removed: ['object']
responses.200.content.application/json.properties.data.items.properties.cancelled_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.cancelling_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.completed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.error_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.errorsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.expired_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.failed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.finalizing_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.in_progress_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.modelType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.output_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.request_countsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.usageType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.has_moreDefault changed: None -> False
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2

POST

Missing Properties (11)
  • requestBody.content.application/json.properties.output_expires_after
  • responses.200.content.application/json.properties.errors.properties.data
  • responses.200.content.application/json.properties.errors.properties.object
  • responses.200.content.application/json.properties.request_counts.properties.completed
  • responses.200.content.application/json.properties.request_counts.properties.failed
  • responses.200.content.application/json.properties.request_counts.properties.total
  • responses.200.content.application/json.properties.usage.properties.input_tokens
  • responses.200.content.application/json.properties.usage.properties.input_tokens_details
  • responses.200.content.application/json.properties.usage.properties.output_tokens
  • responses.200.content.application/json.properties.usage.properties.output_tokens_details
  • responses.200.content.application/json.properties.usage.properties.total_tokens
Schema Issues (15)
PropertyIssues
requestBody.content.application/json.properties.endpointEnum removed: ['/v1/responses', '/v1/chat/completions', '/v1/embeddings', '/v1/completions', '/v1/moderations', '/v1/images/generations', '/v1/images/edits', '/v1/videos']
responses.200.content.application/json.properties.cancelled_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.cancelling_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.completed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.error_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.errorsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.expired_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.failed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.finalizing_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.in_progress_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.modelType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.output_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.request_countsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.usageType removed: ['object']; Union variants added: 2

/batches/{batch_id}​

GET

Missing Properties (10)
  • responses.200.content.application/json.properties.errors.properties.data
  • responses.200.content.application/json.properties.errors.properties.object
  • responses.200.content.application/json.properties.request_counts.properties.completed
  • responses.200.content.application/json.properties.request_counts.properties.failed
  • responses.200.content.application/json.properties.request_counts.properties.total
  • responses.200.content.application/json.properties.usage.properties.input_tokens
  • responses.200.content.application/json.properties.usage.properties.input_tokens_details
  • responses.200.content.application/json.properties.usage.properties.output_tokens
  • responses.200.content.application/json.properties.usage.properties.output_tokens_details
  • responses.200.content.application/json.properties.usage.properties.total_tokens
Schema Issues (14)
PropertyIssues
responses.200.content.application/json.properties.cancelled_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.cancelling_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.completed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.error_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.errorsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.expired_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.failed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.finalizing_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.in_progress_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.modelType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.output_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.request_countsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.usageType removed: ['object']; Union variants added: 2

/batches/{batch_id}/cancel​

POST

Missing Properties (10)
  • responses.200.content.application/json.properties.errors.properties.data
  • responses.200.content.application/json.properties.errors.properties.object
  • responses.200.content.application/json.properties.request_counts.properties.completed
  • responses.200.content.application/json.properties.request_counts.properties.failed
  • responses.200.content.application/json.properties.request_counts.properties.total
  • responses.200.content.application/json.properties.usage.properties.input_tokens
  • responses.200.content.application/json.properties.usage.properties.input_tokens_details
  • responses.200.content.application/json.properties.usage.properties.output_tokens
  • responses.200.content.application/json.properties.usage.properties.output_tokens_details
  • responses.200.content.application/json.properties.usage.properties.total_tokens
Schema Issues (14)
PropertyIssues
responses.200.content.application/json.properties.cancelled_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.cancelling_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.completed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.error_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.errorsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.expired_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.failed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.finalizing_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.in_progress_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.modelType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.output_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.request_countsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.usageType removed: ['object']; Union variants added: 2

Completions​

Score: 56.5% · Issues: 18 · Missing: 2

/completions​

POST

Missing Properties (2)
  • responses.200.content.application/json.properties.system_fingerprint
  • responses.200.content.application/json.properties.usage
Schema Issues (18)
PropertyIssues
requestBody.content.application/json.properties.best_ofType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 1 -> None
requestBody.content.application/json.properties.echoType removed: ['boolean']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: False -> None
requestBody.content.application/json.properties.frequency_penaltyType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 0 -> None
requestBody.content.application/json.properties.logit_biasType removed: ['object']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.logprobsType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.max_tokensType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 16 -> None
requestBody.content.application/json.properties.modelType added: ['string']; Union variants removed: 2
requestBody.content.application/json.properties.nType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 1 -> None
requestBody.content.application/json.properties.presence_penaltyType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 0 -> None
requestBody.content.application/json.properties.promptUnion variants added: 4; Default changed: <|endoftext|> -> None
requestBody.content.application/json.properties.seedType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.stopNullable added (OpenAI non-nullable); Union variants added: 3
requestBody.content.application/json.properties.streamType removed: ['boolean']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: False -> None
requestBody.content.application/json.properties.suffixType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.temperatureType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 1 -> None
requestBody.content.application/json.properties.top_pType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 1 -> None
requestBody.content.application/json.properties.userType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.choices.itemsType removed: ['object']

Vector stores​

Score: 61.3% · Issues: 106 · Missing: 14

/vector_stores​

GET

Missing Properties (2)
  • responses.200.content.application/json.properties.data.items.properties.expires_after.properties.anchor
  • responses.200.content.application/json.properties.data.items.properties.expires_after.properties.days
Schema Issues (12)
PropertyIssues
responses.200.content.application/json.properties.data.itemsType removed: ['object']
responses.200.content.application/json.properties.data.items.properties.expires_afterType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.file_countsType removed: ['object']
responses.200.content.application/json.properties.data.items.properties.metadataType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.objectEnum removed: ['vector_store']; Default changed: None -> vector_store
responses.200.content.application/json.properties.data.items.properties.statusEnum removed: ['expired', 'in_progress', 'completed']; Default changed: None -> completed
responses.200.content.application/json.properties.data.items.properties.usage_bytesDefault changed: None -> 0
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.has_moreDefault changed: None -> False
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectDefault changed: None -> list

POST

Missing Properties (5)
  • requestBody.content.application/json.properties.description
  • requestBody.content.application/json.properties.expires_after.properties.anchor
  • requestBody.content.application/json.properties.expires_after.properties.days
  • responses.200.content.application/json.properties.expires_after.properties.anchor
  • responses.200.content.application/json.properties.expires_after.properties.days
Schema Issues (11)
PropertyIssues
requestBody.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 2
requestBody.content.application/json.properties.expires_afterType removed: ['object']; Union variants added: 2
requestBody.content.application/json.properties.file_idsType removed: ['array']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_afterType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.file_countsType removed: ['object']
responses.200.content.application/json.properties.metadataType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectEnum removed: ['vector_store']; Default changed: None -> vector_store
responses.200.content.application/json.properties.statusEnum removed: ['expired', 'in_progress', 'completed']; Default changed: None -> completed
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

/vector_stores/{vector_store_id}​

DELETE

Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.deletedDefault changed: None -> True
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.deleted']; Default changed: None -> vector_store.deleted

GET

Missing Properties (2)
  • responses.200.content.application/json.properties.expires_after.properties.anchor
  • responses.200.content.application/json.properties.expires_after.properties.days
Schema Issues (7)
PropertyIssues
responses.200.content.application/json.properties.expires_afterType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.file_countsType removed: ['object']
responses.200.content.application/json.properties.metadataType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectEnum removed: ['vector_store']; Default changed: None -> vector_store
responses.200.content.application/json.properties.statusEnum removed: ['expired', 'in_progress', 'completed']; Default changed: None -> completed
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

POST

Missing Properties (2)
  • responses.200.content.application/json.properties.expires_after.properties.anchor
  • responses.200.content.application/json.properties.expires_after.properties.days
Schema Issues (9)
PropertyIssues
requestBody.content.application/json.properties.expires_afterUnion variants added: 2
requestBody.content.application/json.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_afterType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.file_countsType removed: ['object']
responses.200.content.application/json.properties.metadataType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectEnum removed: ['vector_store']; Default changed: None -> vector_store
responses.200.content.application/json.properties.statusEnum removed: ['expired', 'in_progress', 'completed']; Default changed: None -> completed
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

/vector_stores/{vector_store_id}/file_batches​

POST

Missing Properties (1)
  • requestBody.content.application/json.properties.files
Schema Issues (4)
PropertyIssues
requestBody.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.file_countsType removed: ['object']
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.files_batch']; Default changed: None -> vector_store.file_batch
responses.200.content.application/json.properties.statusEnum removed: ['in_progress', 'completed', 'cancelled', 'failed']; Type removed: ['string']; Union variants added: 4

/vector_stores/{vector_store_id}/file_batches/{batch_id}​

GET

Schema Issues (3)
PropertyIssues
responses.200.content.application/json.properties.file_countsType removed: ['object']
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.files_batch']; Default changed: None -> vector_store.file_batch
responses.200.content.application/json.properties.statusEnum removed: ['in_progress', 'completed', 'cancelled', 'failed']; Type removed: ['string']; Union variants added: 4

/vector_stores/{vector_store_id}/file_batches/{batch_id}/cancel​

POST

Schema Issues (3)
PropertyIssues
responses.200.content.application/json.properties.file_countsType removed: ['object']
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.files_batch']; Default changed: None -> vector_store.file_batch
responses.200.content.application/json.properties.statusEnum removed: ['in_progress', 'completed', 'cancelled', 'failed']; Type removed: ['string']; Union variants added: 4

/vector_stores/{vector_store_id}/file_batches/{batch_id}/files​

GET

Schema Issues (11)
PropertyIssues
responses.200.content.application/json.properties.data.itemsType removed: ['object']
responses.200.content.application/json.properties.data.items.properties.attributesType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.chunking_strategyType removed: ['object']; Union variants added: 3; Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.data.items.properties.objectEnum removed: ['vector_store.file']; Default changed: None -> vector_store.file
responses.200.content.application/json.properties.data.items.properties.statusEnum removed: ['in_progress', 'completed', 'cancelled', 'failed']; Type removed: ['string']; Union variants added: 4
responses.200.content.application/json.properties.data.items.properties.usage_bytesDefault changed: None -> 0
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.has_moreDefault changed: None -> False
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectDefault changed: None -> list

/vector_stores/{vector_store_id}/files​

GET

Schema Issues (11)
PropertyIssues
responses.200.content.application/json.properties.data.itemsType removed: ['object']
responses.200.content.application/json.properties.data.items.properties.attributesType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.chunking_strategyType removed: ['object']; Union variants added: 3; Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.data.items.properties.objectEnum removed: ['vector_store.file']; Default changed: None -> vector_store.file
responses.200.content.application/json.properties.data.items.properties.statusEnum removed: ['in_progress', 'completed', 'cancelled', 'failed']; Type removed: ['string']; Union variants added: 4
responses.200.content.application/json.properties.data.items.properties.usage_bytesDefault changed: None -> 0
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.has_moreDefault changed: None -> False
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectDefault changed: None -> list

POST

Schema Issues (7)
PropertyIssues
requestBody.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.attributesType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 3; Union variants removed: 2
responses.200.content.application/json.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.file']; Default changed: None -> vector_store.file
responses.200.content.application/json.properties.statusEnum removed: ['in_progress', 'completed', 'cancelled', 'failed']; Type removed: ['string']; Union variants added: 4
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

/vector_stores/{vector_store_id}/files/{file_id}​

DELETE

Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.deletedDefault changed: None -> True
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.file.deleted']; Default changed: None -> vector_store.file.deleted

GET

Schema Issues (6)
PropertyIssues
responses.200.content.application/json.properties.attributesType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 3; Union variants removed: 2
responses.200.content.application/json.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.file']; Default changed: None -> vector_store.file
responses.200.content.application/json.properties.statusEnum removed: ['in_progress', 'completed', 'cancelled', 'failed']; Type removed: ['string']; Union variants added: 4
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

POST

Schema Issues (7)
PropertyIssues
requestBody.content.application/json.properties.attributesType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.attributesType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 3; Union variants removed: 2
responses.200.content.application/json.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.file']; Default changed: None -> vector_store.file
responses.200.content.application/json.properties.statusEnum removed: ['in_progress', 'completed', 'cancelled', 'failed']; Type removed: ['string']; Union variants added: 4
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

/vector_stores/{vector_store_id}/files/{file_id}/content​

GET

Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.data.itemsType removed: ['object']
responses.200.content.application/json.properties.has_moreDefault changed: None -> False

/vector_stores/{vector_store_id}/search​

POST

Missing Properties (2)
  • requestBody.content.application/json.properties.ranking_options.properties.ranker
  • requestBody.content.application/json.properties.ranking_options.properties.score_threshold
Schema Issues (9)
PropertyIssues
requestBody.content.application/json.properties.filtersUnion variants added: 2
requestBody.content.application/json.properties.max_num_resultsType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.queryUnion variants added: 2
requestBody.content.application/json.properties.ranking_optionsType removed: ['object']; Union variants added: 2
requestBody.content.application/json.properties.rewrite_queryType removed: ['boolean']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.itemsType removed: ['object']
responses.200.content.application/json.properties.data.items.properties.content.itemsType removed: ['object']
responses.200.content.application/json.properties.has_moreDefault changed: None -> False
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.search_results.page']; Default changed: None -> vector_store.search_results.page

Embeddings​

Score: 64.3% · Issues: 5 · Missing: 0

/embeddings​

POST

Schema Issues (5)
PropertyIssues
requestBody.content.application/json.properties.inputUnion variants added: 4
requestBody.content.application/json.properties.modelType added: ['string']; Union variants removed: 2
responses.200.content.application/json.properties.data.itemsType removed: ['object']
responses.200.content.application/json.properties.data.items.properties.embeddingType removed: ['array']; Union variants added: 2
responses.200.content.application/json.properties.usageType removed: ['object']

Files​

Score: 66.7% · Issues: 8 · Missing: 6

/files​

GET

Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.data.items.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.purposeEnum values removed: ['assistants_output', 'batch_output', 'fine-tune', 'fine-tune-results', 'vision', 'user_data']

POST

Missing Properties (4)
  • requestBody.content.multipart/form-data.properties.expires_after.properties.anchor
  • requestBody.content.multipart/form-data.properties.expires_after.properties.seconds
  • responses.200.content.application/json.properties.status
  • responses.200.content.application/json.properties.status_details
Schema Issues (4)
PropertyIssues
requestBody.content.multipart/form-data.properties.expires_afterType removed: ['object']; Union variants added: 2
requestBody.content.multipart/form-data.properties.purposeEnum values removed: ['fine-tune', 'vision', 'user_data', 'evals']
responses.200.content.application/json.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.purposeEnum values removed: ['assistants_output', 'batch_output', 'fine-tune', 'fine-tune-results', 'vision', 'user_data']

/files/{file_id}​

GET

Missing Properties (2)
  • responses.200.content.application/json.properties.status
  • responses.200.content.application/json.properties.status_details
Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.purposeEnum values removed: ['assistants_output', 'batch_output', 'fine-tune', 'fine-tune-results', 'vision', 'user_data']

Models​

Score: 66.7% · Issues: 0 · Missing: 5

/models​

GET

Missing Properties (1)
  • responses.200.content.application/json.properties.object

/models/{model}​

GET

Missing Properties (4)
  • responses.200.content.application/json.properties.created
  • responses.200.content.application/json.properties.id
  • responses.200.content.application/json.properties.object
  • responses.200.content.application/json.properties.owned_by

Chat​

Score: 87.1% · Issues: 33 · Missing: 19

/chat/completions​

GET

Missing Properties (5)
  • parameters.query.metadata
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.completion_tokens_details.properties.accepted_prediction_tokens
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.completion_tokens_details.properties.audio_tokens
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.completion_tokens_details.properties.rejected_prediction_tokens
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.prompt_tokens_details.properties.audio_tokens
Schema Issues (9)
PropertyIssues
responses.200.content.application/json.properties.data.itemsType removed: ['object']
responses.200.content.application/json.properties.data.items.properties.choices.itemsType removed: ['object']
responses.200.content.application/json.properties.data.items.properties.choices.items.properties.messageType removed: ['object']
responses.200.content.application/json.properties.data.items.properties.choices.items.properties.message.properties.function_callType removed: ['object']
responses.200.content.application/json.properties.data.items.properties.choices.items.properties.message.properties.tool_calls.itemsUnion variants added: 2
responses.200.content.application/json.properties.data.items.properties.usageType removed: ['object']
responses.200.content.application/json.properties.data.items.properties.usage.properties.completion_tokens_detailsType removed: ['object']
responses.200.content.application/json.properties.data.items.properties.usage.properties.prompt_tokens_detailsType removed: ['object']
responses.200.content.application/json.properties.objectDefault changed: list -> None

POST

Missing Properties (10)
  • responses.200.content.application/json.properties.usage.properties.completion_tokens_details.properties.accepted_prediction_tokens
  • responses.200.content.application/json.properties.usage.properties.completion_tokens_details.properties.audio_tokens
  • responses.200.content.application/json.properties.usage.properties.completion_tokens_details.properties.rejected_prediction_tokens
  • responses.200.content.application/json.properties.usage.properties.prompt_tokens_details.properties.audio_tokens
  • responses.200.content.text/event-stream.properties.choices.items.properties.delta.properties.function_call
  • responses.200.content.text/event-stream.properties.choices.items.properties.delta.properties.tool_calls.items.properties.index
  • responses.200.content.text/event-stream.properties.usage.properties.completion_tokens_details.properties.accepted_prediction_tokens
  • responses.200.content.text/event-stream.properties.usage.properties.completion_tokens_details.properties.audio_tokens
  • responses.200.content.text/event-stream.properties.usage.properties.completion_tokens_details.properties.rejected_prediction_tokens
  • responses.200.content.text/event-stream.properties.usage.properties.prompt_tokens_details.properties.audio_tokens
Schema Issues (17)
PropertyIssues
responses.200.content.application/json.properties.choices.itemsType removed: ['object']
responses.200.content.application/json.properties.choices.items.properties.messageType removed: ['object']
responses.200.content.application/json.properties.choices.items.properties.message.properties.function_callType removed: ['object']
responses.200.content.application/json.properties.choices.items.properties.message.properties.tool_calls.itemsUnion variants added: 2
responses.200.content.application/json.properties.usageType removed: ['object']
responses.200.content.application/json.properties.usage.properties.completion_tokens_detailsType removed: ['object']
responses.200.content.application/json.properties.usage.properties.prompt_tokens_detailsType removed: ['object']
responses.200.content.text/event-stream.properties.choices.itemsType removed: ['object']
responses.200.content.text/event-stream.properties.choices.items.properties.deltaType removed: ['object']
responses.200.content.text/event-stream.properties.choices.items.properties.delta.properties.tool_calls.itemsType removed: ['object']
responses.200.content.text/event-stream.properties.choices.items.properties.delta.properties.tool_calls.items.properties.functionType removed: ['object']
responses.200.content.text/event-stream.properties.choices.items.properties.logprobsType removed: ['object']
responses.200.content.text/event-stream.properties.choices.items.properties.logprobs.properties.contentType removed: ['array']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.text/event-stream.properties.choices.items.properties.logprobs.properties.refusalType removed: ['array']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.text/event-stream.properties.usageType removed: ['object']
responses.200.content.text/event-stream.properties.usage.properties.completion_tokens_detailsType removed: ['object']
responses.200.content.text/event-stream.properties.usage.properties.prompt_tokens_detailsType removed: ['object']

/chat/completions/{completion_id}​

GET

Missing Properties (4)
  • responses.200.content.application/json.properties.usage.properties.completion_tokens_details.properties.accepted_prediction_tokens
  • responses.200.content.application/json.properties.usage.properties.completion_tokens_details.properties.audio_tokens
  • responses.200.content.application/json.properties.usage.properties.completion_tokens_details.properties.rejected_prediction_tokens
  • responses.200.content.application/json.properties.usage.properties.prompt_tokens_details.properties.audio_tokens
Schema Issues (7)
PropertyIssues
responses.200.content.application/json.properties.choices.itemsType removed: ['object']
responses.200.content.application/json.properties.choices.items.properties.messageType removed: ['object']
responses.200.content.application/json.properties.choices.items.properties.message.properties.function_callType removed: ['object']
responses.200.content.application/json.properties.choices.items.properties.message.properties.tool_calls.itemsUnion variants added: 2
responses.200.content.application/json.properties.usageType removed: ['object']
responses.200.content.application/json.properties.usage.properties.completion_tokens_detailsType removed: ['object']
responses.200.content.application/json.properties.usage.properties.prompt_tokens_detailsType removed: ['object']

Responses​

Score: 87.1% · Issues: 29 · Missing: 0

/responses​

POST

Schema Issues (29)
PropertyIssuesTested
requestBody.content.application/json.properties.includeType removed: ['array']; Nullable added (OpenAI non-nullable); Union variants added: 2No
requestBody.content.application/json.properties.inputUnion variants added: 2; Union variants removed: 2Yes
requestBody.content.application/json.properties.modelType added: ['string']; Nullable removed (OpenAI nullable); Union variants removed: 2Yes
requestBody.content.application/json.properties.parallel_tool_callsDefault changed: None -> TrueYes
requestBody.content.application/json.properties.reasoningUnion variants added: 1; Union variants removed: 1Yes
requestBody.content.application/json.properties.service_tierUnion variants added: 2Yes
requestBody.content.application/json.properties.storeType removed: ['boolean']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: None -> TrueYes
requestBody.content.application/json.properties.streamType removed: ['boolean']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: None -> FalseYes
requestBody.content.application/json.properties.stream_optionsUnion variants added: 1; Union variants removed: 1No
requestBody.content.application/json.properties.textUnion variants added: 1; Union variants removed: 1No
requestBody.content.application/json.properties.tool_choiceUnion variants added: 2; Union variants removed: 1Yes
requestBody.content.application/json.properties.truncationUnion variants added: 2Yes
responses.200.content.application/json.properties.errorUnion variants added: 1; Union variants removed: 1Yes
responses.200.content.application/json.properties.frequency_penaltyType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2No
responses.200.content.application/json.properties.incomplete_detailsUnion variants added: 1; Union variants removed: 1Yes
responses.200.content.application/json.properties.metadataUnion variants added: 2Yes
responses.200.content.application/json.properties.objectDefault changed: response -> NoneNo
responses.200.content.application/json.properties.output.itemsUnion variants added: 8; Union variants removed: 4Yes
responses.200.content.application/json.properties.parallel_tool_callsType removed: ['boolean']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: None -> TrueYes
responses.200.content.application/json.properties.presence_penaltyType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2No
responses.200.content.application/json.properties.reasoningUnion variants added: 1; Union variants removed: 1Yes
responses.200.content.application/json.properties.service_tierType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2Yes
responses.200.content.application/json.properties.temperatureType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2Yes
responses.200.content.application/json.properties.tool_choiceUnion variants added: 3Yes
responses.200.content.application/json.properties.toolsType removed: ['array']; Nullable added (OpenAI non-nullable); Union variants added: 2Yes
responses.200.content.application/json.properties.top_logprobsType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2Yes
responses.200.content.application/json.properties.top_pType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2Yes
responses.200.content.application/json.properties.truncationUnion variants added: 2Yes
responses.200.content.application/json.properties.usageUnion variants added: 1; Union variants removed: 1Yes

Conversations​

Score: 98.8% · Issues: 22 · Missing: 4

/conversations​

POST

Schema Issues (3)
PropertyIssuesTested
requestBody.content.application/json.properties.metadataUnion variants added: 1; Union variants removed: 1Yes
responses.200.content.application/json.properties.metadataUnion variants added: 2Yes
responses.200.content.application/json.properties.objectDefault changed: conversation -> NoneNo

/conversations/{conversation_id}​

DELETE

Schema Issues (2)
PropertyIssuesTested
responses.200.content.application/json.properties.deletedDefault changed: None -> TrueNo
responses.200.content.application/json.properties.objectEnum removed: ['conversation.deleted']No

GET

Schema Issues (2)
PropertyIssuesTested
responses.200.content.application/json.properties.metadataUnion variants added: 2Yes
responses.200.content.application/json.properties.objectDefault changed: conversation -> NoneNo

POST

Schema Issues (3)
PropertyIssuesTested
requestBody.content.application/json.properties.metadataType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2Yes
responses.200.content.application/json.properties.metadataUnion variants added: 2Yes
responses.200.content.application/json.properties.objectDefault changed: conversation -> NoneNo

/conversations/{conversation_id}/items​

GET

Schema Issues (5)
PropertyIssuesTested
responses.200.content.application/json.properties.data.itemsUnion variants added: 10; Union variants removed: 25No
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2No
responses.200.content.application/json.properties.has_moreDefault changed: None -> FalseNo
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2No
responses.200.content.application/json.properties.objectEnum removed: ['list']; Default changed: None -> listNo

POST

Missing Properties (1)
  • parameters.query.include
Schema Issues (6)
PropertyIssuesTested
requestBody.content.application/json.properties.items.itemsUnion variants added: 10; Union variants removed: 3No
responses.200.content.application/json.properties.data.itemsUnion variants added: 10; Union variants removed: 25No
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2No
responses.200.content.application/json.properties.has_moreDefault changed: None -> FalseNo
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2No
responses.200.content.application/json.properties.objectEnum removed: ['list']; Default changed: None -> listNo

/conversations/{conversation_id}/items/{item_id}​

DELETE

Missing Properties (2)
  • responses.200.content.application/json.properties.created_at
  • responses.200.content.application/json.properties.metadata
Schema Issues (1)
PropertyIssuesTested
responses.200.content.application/json.properties.objectEnum removed: ['conversation']; Default changed: conversation -> conversation.item.deletedNo

GET

Missing Properties (1)
  • parameters.query.include

How to Improve Conformance​

To improve conformance scores:

  1. Fix Schema Issues: Update Pydantic models in src/llama_stack_api/ to match OpenAI's schema
  2. Add Missing Properties: Implement missing fields in response models
  3. Add Missing Endpoints: Implement endpoints listed in the Missing Endpoints section

Run the coverage analyzer to check your progress:

python scripts/openai_coverage.py --update

Then regenerate this documentation:

python scripts/generate_openai_coverage_docs.py