[FEATURE]: Add Omniroute provider with dynamic model discovery #25947

Activity

1. github-actions

   assigned

   kitlangton

   on May 6, 2026
2. github-actions commented on May 6, 2026

   

   github-actionsBot

   on May 6, 2026 – with GitHub Actions

   Contributor

   More actions

   This issue doesn't fully meet our contributing guidelines.

   What needs to be fixed:

   • Feature request titles must start with [FEATURE]: (e.g. [FEATURE]: Add Omniroute provider with dynamic model discovery)
   • The feature request template requires checking the verification checkbox confirming the feature hasn't been suggested before

   Please edit this issue to address the above within 2 hours, or it will be automatically closed.

   ---

   This issue might be a duplicate of existing issues. Please check:

   • Auto-discover models from OpenAI-compatible provider endpoints #6231: Auto-discover models from OpenAI-compatible provider endpoints (same core request: dynamic /v1/models discovery for OpenAI-compatible providers)
   • [FEATURE]: Detect model(s) using the /v1/models route on custom provider #25624: [FEATURE]: Detect model(s) using the /v1/models route on custom provider (very similar: auto-detect models via /v1/models on custom providers)
   • [Feature Request]: Allow dynamic model passthrough for custom OpenAI-compatible providers without explicit declaration #18219: [Feature Request]: Allow dynamic model passthrough for custom OpenAI-compatible providers without explicit declaration (closely related: dynamic model support for custom OpenAI-compatible providers)

   If you believe this was flagged incorrectly, please let a maintainer know.
3. github-actions

   added

   needs:complianceThis means the issue will auto-close after 2 hours.This means the issue will auto-close after 2 hours.

   on May 6, 2026
4. disonjer

   changed the title [-]Add Omniroute provider with dynamic model discovery[/-] [+][FEATURE]: Add Omniroute provider with dynamic model discovery[/+] on May 6, 2026
5. github-actions

   removed

   needs:complianceThis means the issue will auto-close after 2 hours.This means the issue will auto-close after 2 hours.

   on May 6, 2026
6. github-actions commented on May 6, 2026

   

   github-actionsBot

   on May 6, 2026 – with GitHub Actions

   Contributor

   More actions

   Thank you for updating the issue! It now follows the contributing guidelines and looks good. We appreciate the clear description and the context you provided about Omniroute.
7. XidaoApi commented on May 9, 2026

   

   XidaoApi

   on May 9, 2026

   More actions

   Dynamic model discovery is the right approach for OpenAI-compatible gateways. Static model lists break the moment the gateway adds or removes models. Here's a pattern that works reliably across diverse backends:

   1. Probe /v1/models with graceful fallback

   async def discover_models(base_url: str, api_key: str, timeout: float = 5.0) -> list[str]:
       """Discover available models from an OpenAI-compatible endpoint."""
       try:
           async with httpx.AsyncClient(timeout=timeout) as client:
               resp = await client.get(
                   f"{base_url.rstrip('/')}/v1/models",
                   headers={"Authorization": f"Bearer {api_key}"}
               )
               if resp.status_code == 200:
                   data = resp.json()
                   return [m["id"] for m in data.get("data", [])]
       except (httpx.TimeoutException, httpx.ConnectError):
           pass
       return []  # Caller falls back to static list

   2. Cache with TTL and background refresh

   Model lists rarely change mid-session. Cache for 5-15 minutes with a background refresh:

   class ModelCache:
       def __init__(self, ttl_seconds=300):
           self._cache: dict[str, tuple[float, list[str]]] = {}
           self._ttl = ttl_seconds
   
       async def get_models(self, base_url, api_key) -> list[str]:
           key = base_url.rstrip("/")
           now = time.time()
           if key in self._cache:
               ts, models = self._cache[key]
               if now - ts < self._ttl:
                   return models
           models = await discover_models(base_url, api_key)
           if models:
               self._cache[key] = (now, models)
           return models

   3. Capability inference from model IDs

   When the /v1/models response doesn't include capability metadata, infer from naming patterns:

   • *instruct* -> chat/completion
   • *embed* -> embeddings only
   • *vision* or *omni* -> multimodal
   • *tts* / *whisper* -> audio

   This won't be perfect but handles 90% of real-world model names.

   4. Health check on startup

   Run a lightweight /v1/models probe when the provider is first configured and surface errors immediately in the UI rather than waiting for the first chat request to fail.

   This pattern is what many multi-provider tools use (including gateway services) — it eliminates the "model not found" class of errors that users hit when gateways update their model catalog.

   🚀React with 🚀1Reacted by Container
8. github-actions commented on Jul 9, 2026

   

   github-actionsBot

   on Jul 9, 2026 – with GitHub Actions

   Contributor

   More actions

   To stay organized issues are automatically closed after 60 days of no activity. If the issue is still relevant please open a new one.
9. github-actions

   closed this as not plannedon Jul 9, 2026