Skip to main content

Creating External Providers

Configuration​

To enable external providers, you need to add module into your build yaml, allowing OGX to install the required package corresponding to the external provider.

an example entry in your build.yaml should look like:

- provider_type: remote::ramalama
  module: ramalama_stack

Provider Types​

OGX supports two types of external providers:

  1. Remote Providers: Providers that communicate with external services (e.g., cloud APIs)
  2. Inline Providers: Providers that run locally within the OGX process

Provider Specification (Common between inline and remote providers)​

  • provider_type: The type of the provider to be installed (remote or inline). eg. remote::ollama
  • api: The API for this provider, eg. inference
  • config_class: The full path to the configuration class
  • module: The Python module containing the provider implementation
  • optional_api_dependencies: List of optional OGX APIs that this provider can use
  • api_dependencies: List of OGX APIs that this provider depends on
  • provider_data_validator: Optional validator for provider data.
  • pip_packages: List of Python packages required by the provider

Remote Provider Specification​

Remote providers are used when you need to communicate with external services. Here's an example for a custom Ollama provider:

adapter_type: custom_ollama
provider_type: "remote::ollama"
pip_packages:
- ollama
- aiohttp
config_class: ogx_ollama_provider.config.OllamaImplConfig
module: ogx_ollama_provider
api_dependencies: []
optional_api_dependencies: []

Remote Provider Configuration​

  • adapter_type: A unique identifier for this adapter, eg. ollama

Inline Provider Specification​

Inline providers run locally within the OGX process. Here's an example for a custom vector store provider:

module: ogx_vector_provider
provider_type: inline::ogx_vector_provider
config_class: ogx_vector_provider.config.VectorStoreConfig
pip_packages:
  - faiss-cpu
  - numpy
api_dependencies:
  - inference
optional_api_dependencies:
  - vector_io
provider_data_validator: ogx_vector_provider.validator.VectorStoreValidator
container_image: custom-vector-store:latest  # optional

Inline Provider Fields​

  • container_image: Optional container image to use instead of pip packages

Required Fields​

All Providers​

All providers must contain a get_provider_spec function in their provider module. This is a standardized structure that OGX expects and is necessary for getting things such as the config class. The get_provider_spec method returns a structure identical to the adapter. An example function may look like:

from ogx_api.providers.datatypes import (
    ProviderSpec,
    Api,
    RemoteProviderSpec,
)


def get_provider_spec() -> ProviderSpec:
    return RemoteProviderSpec(
        api=Api.inference,
        adapter_type="ramalama",
        pip_packages=["ramalama>=0.8.5", "pymilvus"],
        config_class="ramalama_stack.config.RamalamaImplConfig",
        module="ramalama_stack",
    )

Remote Providers​

Remote providers must expose a get_adapter_impl() function in their module that takes two arguments:

  1. config: An instance of the provider's config class
  2. deps: A dictionary of API dependencies

This function must return an instance of the provider's adapter class that implements the required protocol for the API.

Example:

async def get_adapter_impl(
    config: OllamaImplConfig, deps: Dict[Api, Any]
) -> OllamaInferenceAdapter:
    return OllamaInferenceAdapter(config)

Inline Providers​

Inline providers must expose a get_provider_impl() function in their module that takes two arguments:

  1. config: An instance of the provider's config class
  2. deps: A dictionary of API dependencies

Example:

async def get_provider_impl(
    config: VectorStoreConfig, deps: Dict[Api, Any]
) -> VectorStoreImpl:
    impl = VectorStoreImpl(config, deps[Api.inference])
    await impl.initialize()
    return impl

Dependencies​

The provider package must be installed on the system. For example:

$ uv pip show ogx-ollama-provider
Name: ogx-ollama-provider
Version: 0.1.0
Location: /path/to/venv/lib/python3.10/site-packages

Best Practices​

  1. Package Naming: Use the prefix ogx-provider- for your provider packages to make them easily identifiable.

  2. Version Management: Keep your provider package versioned and compatible with the OGX version you're using.

  3. Dependencies: Only include the minimum required dependencies in your provider package.

  4. Documentation: Include clear documentation in your provider package about:

    • Installation requirements
    • Configuration options
    • Usage examples
    • Any limitations or known issues

  5. Testing: Include tests in your provider package to ensure it works correctly with OGX. You can refer to the integration tests guide for more information. Execute the test for the Provider type you are developing.

Troubleshooting​

If your external provider isn't being loaded:

  1. Check that module points to a published pip package with a top level provider module including get_provider_spec.
  2. Verify that the YAML files are properly formatted.
  3. Ensure all required Python packages are installed.
  4. Check the OGX server logs for any error messages - turn on debug logging to get more information using OGX_LOGGING=all=debug.

Examples​

How to create an external provider module​

If you are creating a new external provider called ogx-provider-ollama here is how you would set up the package properly:

  1. First, create the provider package:
mkdir -p ogx-provider-ollama
cd ogx-provider-ollama
git init
uv init
  1. Edit pyproject.toml:
[project]
name = "ogx-provider-ollama"
version = "0.1.0"
description = "Ollama provider for OGX"
requires-python = ">=3.12"
dependencies = ["ogx", "pydantic", "ollama", "aiohttp"]
  1. Install the provider:
uv pip install -e .
  1. Edit provider.py

provider.py must be updated to contain get_provider_spec. This is used by ogx to install the provider.

def get_provider_spec() -> ProviderSpec:
    return RemoteProviderSpec(
        api=Api.inference,
        adapter_type="ogx-provider-ollama",
        pip_packages=["ollama", "aiohttp"],
        config_class="ogx_provider_ollama.config.OllamaImplConfig",
        module="ogx_provider_ollama",
    )
  1. Implement the provider as outlined above with get_provider_impl or get_adapter_impl, etc.

Example using module: ramalama-stack​

ramalama-stack is a recognized external provider that supports installation via module.

To install OGX with this external provider a user can provider the following config.yaml:

version: 2
distro_name: ramalama
apis:
- inference
providers:
  inference:
  - provider_id: ramalama
    provider_type: remote::ramalama
    module: ramalama_stack==0.3.0a0
    config: {}
server:
  port: 8321

No other steps are required beyond installing dependencies with ogx list-deps <distro> | xargs -L1 uv pip install and then running ogx stack run. The CLI will use module to install the provider dependencies, retrieve the spec, etc.

The provider will now be available in OGX with the type remote::ramalama.