Repository navigation
Open
Labels
P4Backlog - Very lowBacklog - Very lowenhancementNew feature or requestNew feature or requestperformancePerformance optimizationPerformance optimization
Description
Summary
Provider.state costs about 23 ms on every run. Most of that is two things:
- A
structuredCloneof the whole model catalog: 19 providers, 1,376 models. - An eager
import("@gitlab/gitlab-ai-provider"), used only to build aUser-Agentstring. It runs even without GitLab credentials.
Fixing both halves the cost: about −23 ms in a bare run, −17 ms with MCP.
Change
- gitlab loader: return
{ autoload: false }before the import when there's no key from OAuth, an API key orGITLAB_TOKEN. Same pattern as the Bedrock loader. - Copy on take: copy built-in providers only when the state takes them.
ModelCatalog.shared()returns the entries uncopied;ModelCatalog.provider(id)returns a copy of one provider.provider.tsgets a smallproviderDatabase()helper withpeek(read-only),take(copy on first use, memoised),setandids.idskeeps the catalog order.- Callers that mutate or alias a provider (
mergeProvider, config extend, auth plugins) usetake. - Callers that only read (the env scan, the custom loaders) use
peek.CustomLoaderis now typedReadonly<Info>.
The copy can't simply be dropped: mergeDeep aliases models, and four places mutate it.
Measurements
Measured on main @ e88ed7f432 with Bun 1.4.3. Each run was a fresh container, in-process time from process start to the first model request, median of 20 runs.
| Setup | main |
|---|---|
| (a) bare: no MCP, no skills | 287 ms |
| (b) with a remote MCP server, skills and a git repo | 482 ms |
| (b′) as (b), plus 100 ms per new TCP+TLS connection to the MCP host | 678 ms |
| Setup | Saving |
|---|---|
| (a) | −23 ms |
| (b) | −17 ms |
aictrl models output is byte-identical across 8 provider and config scenarios.
Risks
- gitlab edge case: a third-party auth plugin for
gitlabwith awellknowncredential and no token would no longer get the loader's options. No such plugin exists in-tree. - Read-only contract: loaders now see shared entries. A future loader that mutates its input would corrupt the catalog. The isolation test catches it.
Tests
gitlab-duo.test.ts: the SDK isn't imported without credentials (fails onmain), and is imported withGITLAB_TOKEN.catalog.test.ts: after a full state load with config, env, the Codex plugin andlist()mutation,shared()still deep-equalscatalog.json. Mutation-checked.provider.test.ts:list()order and thedefaultModel()choice are unchanged.bun test test/provider: 347 pass.
Prototype patch
diff --git a/packages/cli/src/provider/catalog.ts b/packages/cli/src/provider/catalog.ts
index 41dd0cf6ec..b14e5b1198 100644
--- a/packages/cli/src/provider/catalog.ts
+++ b/packages/cli/src/provider/catalog.ts
@@ -18,10 +18,26 @@ export namespace ModelCatalog {
}
const entries = catalog as unknown as Record<string, Entry>
+ const builtIn: Record<string, Provider.Info> = mapValues(entries, ({ npm: _npm, api: _api, ...info }) => info)
+
+ /**
+ * The built-in providers themselves, shared by every caller. Read them without changing
+ * them; copy a provider with `provider` before changing it.
+ */
+ export function shared(): Readonly<Record<string, Provider.Info>> {
+ return builtIn
+ }
+
+ /** Returns a fresh copy of one built-in provider, because provider state mutates the providers it loads. */
+ export function provider(providerID: string): Provider.Info | undefined {
+ const info = builtIn[providerID]
+ if (!info) return undefined
+ return structuredClone(info)
+ }
/** Returns fresh copies, because provider state mutates the providers it loads. */
export function providers(): Record<string, Provider.Info> {
- return mapValues(entries, ({ npm: _npm, api: _api, ...info }) => structuredClone(info))
+ return mapValues(builtIn, (info) => structuredClone(info))
}
export function sdkDefaults(providerID: string): Partial<SDKDefaults> {
diff --git a/packages/cli/src/provider/provider.ts b/packages/cli/src/provider/provider.ts
index a05446f568..cf0ea8f964 100644
--- a/packages/cli/src/provider/provider.ts
+++ b/packages/cli/src/provider/provider.ts
@@ -108,7 +108,8 @@ export namespace Provider {
}
type CustomModelLoader = (sdk: any, modelID: string, options?: Record<string, any>) => Promise<any>
- type CustomLoader = (provider: Info) => Promise<{
+ /** Receives the provider without a copy, possibly the shared catalog entry: read it, never change it. */
+ type CustomLoader = (provider: Readonly<Info>) => Promise<{
autoload: boolean
getModel?: CustomModelLoader
options?: Record<string, any>
@@ -443,6 +444,9 @@ export namespace Provider {
if (auth?.type === "api") return auth.key
return Env.get("GITLAB_TOKEN")
})()
+ // Without a key GitLab does not autoload and its options go unused, so skip importing
+ // its SDK, which bundles several API clients.
+ if (!apiKey) return { autoload: false }
const config = await Config.get()
const providerConfig = config.provider?.["gitlab"]
@@ -677,10 +681,36 @@ export namespace Provider {
return error
}
+ /**
+ * The providers one state works with: the built-in catalog plus providers declared in config.
+ * Loading a provider mutates its provider and model objects, and the catalog is shared, so a
+ * built-in provider is copied the first time this state takes it. Providers that are only
+ * inspected, such as those without credentials, are never copied.
+ */
+ function providerDatabase() {
+ const builtIn = ModelCatalog.shared()
+ const owned: Record<string, Info> = {}
+ return {
+ /** Built-in providers in catalog order, then providers only config declares. */
+ ids: () => Object.keys({ ...builtIn, ...owned }),
+ /** Read-only: the provider this state would load, without copying it. */
+ peek: (providerID: string): Info | undefined => owned[providerID] ?? builtIn[providerID],
+ /** The provider as this state's own copy, safe to mutate. */
+ take(providerID: string): Info | undefined {
+ const provider = owned[providerID] ?? ModelCatalog.provider(providerID)
+ if (provider) owned[providerID] = provider
+ return provider
+ },
+ set(providerID: string, provider: Info) {
+ owned[providerID] = provider
+ },
+ }
+ }
+
const state = Instance.state(async () => {
using _ = log.time("state")
const config = await Config.get()
- const database = ModelCatalog.providers()
+ const database = providerDatabase()
const disabled = new Set(config.disabled_providers ?? [])
const enabled = config.enabled_providers ? new Set(config.enabled_providers) : null
@@ -710,7 +740,7 @@ export namespace Provider {
providers[providerID] = mergeDeep(existing, provider)
return
}
- const match = database[providerID]
+ const match = database.take(providerID)
if (!match) return
// @ts-expect-error
providers[providerID] = mergeDeep(match, provider)
@@ -718,7 +748,7 @@ export namespace Provider {
// extend database from config
for (const [providerID, provider] of configProviders) {
- const existing = database[providerID]
+ const existing = database.take(providerID)
const sdkDefaults = ModelCatalog.sdkDefaults(providerID)
const parsed: Info = {
id: providerID,
@@ -797,13 +827,14 @@ export namespace Provider {
)
parsed.models[modelID] = parsedModel
}
- database[providerID] = parsed
+ database.set(providerID, parsed)
}
// load env
const env = Env.all()
- for (const [providerID, provider] of Object.entries(database)) {
+ for (const providerID of database.ids()) {
if (disabled.has(providerID)) continue
+ const provider = database.peek(providerID)!
const apiKey = provider.env.map((item) => env[item]).find(Boolean)
if (!apiKey) continue
mergeProvider(providerID, {
@@ -828,10 +859,11 @@ export namespace Provider {
const providerID = plugin.auth.provider
if (disabled.has(providerID)) continue
if (!plugin.auth.loader) continue
- const provider = database[providerID]
- if (!provider) continue
+ if (!database.peek(providerID)) continue
if (!(await Auth.get(providerID))) continue
+ // Auth plugins may change the provider's models, so they get this state's own copy.
+ const provider = database.take(providerID)!
const options = await plugin.auth.loader(() => Auth.get(providerID) as any, provider)
const opts = options ?? {}
const patch: Partial<Info> = providers[providerID] ? { options: opts } : { source: "custom", options: opts }
@@ -840,7 +872,7 @@ export namespace Provider {
for (const [providerID, fn] of Object.entries(CUSTOM_LOADERS)) {
if (disabled.has(providerID)) continue
- const data = database[providerID]
+ const data = database.peek(providerID)
if (!data) {
log.error("Provider does not exist in the model catalog " + providerID)
continue
diff --git a/packages/cli/test/provider/catalog.test.ts b/packages/cli/test/provider/catalog.test.ts
index 1507107eef..e984fe272f 100644
--- a/packages/cli/test/provider/catalog.test.ts
+++ b/packages/cli/test/provider/catalog.test.ts
@@ -248,3 +248,76 @@ describe("provider registry", () => {
})
})
})
+
+describe("provider state and the shared catalog", () => {
+ const catalogPath = path.join(import.meta.dir, "../../src/provider/catalog.json")
+
+ /** The catalog as it ships, read from disk so a mutated in-memory copy cannot hide a leak. */
+ async function shippedProviders(): Promise<Record<string, Provider.Info>> {
+ const entries = JSON.parse(await Bun.file(catalogPath).text()) as Record<string, Provider.Info & { npm: string }>
+ return Object.fromEntries(
+ Object.entries(entries).map(([providerID, { npm: _npm, api: _api, ...info }]: [string, any]) => [
+ providerID,
+ info,
+ ]),
+ )
+ }
+
+ test("provider returns a copy of one built-in provider", () => {
+ const copy = ModelCatalog.provider("anthropic")!
+ expect(copy).toEqual(ModelCatalog.shared()["anthropic"])
+ delete copy.models["claude-haiku-4-5"]
+ expect(ModelCatalog.shared()["anthropic"].models["claude-haiku-4-5"]).toBeDefined()
+ expect(ModelCatalog.provider("not-built-in")).toBeUndefined()
+ })
+
+ test("loading, filtering, extending and plugin changes never reach the shared catalog", async () => {
+ await using tmp = await tmpdir({
+ init: async (dir) => {
+ await Bun.write(
+ path.join(dir, "aictrl.json"),
+ JSON.stringify({
+ $schema: "https://aictrl.ai/config.json",
+ provider: {
+ // Config extends a built-in provider's models map and filters it.
+ anthropic: {
+ blacklist: ["claude-haiku-4-5"],
+ options: { timeout: 1 },
+ models: { "config-only-model": { name: "Config only" } },
+ },
+ },
+ }),
+ )
+ },
+ })
+ // The Codex auth plugin deletes OpenAI models and rewrites their costs on the provider it is given.
+ const authPath = path.join(Global.Path.data, "auth.json")
+ const savedAuth = await Bun.file(authPath)
+ .text()
+ .catch(() => "{}")
+ await Bun.write(
+ authPath,
+ JSON.stringify({ openai: { type: "oauth", access: "a", refresh: "r", expires: Date.now() + 3600000 } }),
+ )
+ try {
+ await Instance.provide({
+ directory: tmp.path,
+ init: async () => {
+ Env.set("ANTHROPIC_API_KEY", "key")
+ },
+ fn: async () => {
+ const providers = await Provider.list()
+ expect(providers["anthropic"].models["claude-haiku-4-5"]).toBeUndefined()
+ expect(providers["anthropic"].models["config-only-model"]).toBeDefined()
+ expect(providers["openai"].models["gpt-5-nano"]).toBeUndefined()
+ expect(providers["openai"].models["gpt-6.1-sol"].cost.input).toBe(0)
+ // Callers of list() may change what it returns as well.
+ providers["anthropic"].name = "changed"
+ },
+ })
+ } finally {
+ await Bun.write(authPath, savedAuth)
+ }
+ expect(ModelCatalog.shared()).toEqual(await shippedProviders())
+ })
+})
diff --git a/packages/cli/test/provider/gitlab-duo.test.ts b/packages/cli/test/provider/gitlab-duo.test.ts
index 4315016dac..5f744c59ac 100644
--- a/packages/cli/test/provider/gitlab-duo.test.ts
+++ b/packages/cli/test/provider/gitlab-duo.test.ts
@@ -1,4 +1,4 @@
-import { test, expect } from "bun:test"
+import { describe, test, expect } from "bun:test"
import path from "path"
import { tmpdir } from "../fixture/fixture"
@@ -260,3 +260,48 @@ test("GitLab Duo: has multiple agentic chat models available", async () => {
},
})
})
+
+describe("GitLab Duo: SDK import", () => {
+ const entry = path.resolve(import.meta.dir, "../../src/index.ts")
+
+ /** Runs `aictrl models anthropic` in a fresh process and reports whether it evaluated the GitLab SDK. */
+ async function importsGitLabSDK(env: Record<string, string>) {
+ await using tmp = await tmpdir()
+ const probe = path.join(tmp.path, "probe.ts")
+ await Bun.write(
+ probe,
+ `process.on("exit", () => console.error("GITLAB_SDK_LOADED=" + Object.keys(require.cache).some((id) => id.includes("gitlab-ai-provider"))))`,
+ )
+ const proc = Bun.spawn(["bun", "--preload", probe, "--conditions=browser", entry, "models", "anthropic"], {
+ cwd: tmp.path,
+ // Config, auth, cache and state live under the fresh tmpdir, so no stored GitLab credential applies.
+ env: {
+ ...process.env,
+ GITLAB_TOKEN: "",
+ XDG_DATA_HOME: path.join(tmp.path, "data"),
+ XDG_CACHE_HOME: path.join(tmp.path, "cache"),
+ XDG_CONFIG_HOME: path.join(tmp.path, "config"),
+ XDG_STATE_HOME: path.join(tmp.path, "state"),
+ AICTRL_TEST_HOME: tmp.path,
+ AICTRL_TEST_MANAGED_CONFIG_DIR: path.join(tmp.path, "managed"),
+ ANTHROPIC_API_KEY: "test-key",
+ ...env,
+ },
+ stdout: "pipe",
+ stderr: "pipe",
+ })
+ const [stderr, exit] = await Promise.all([new Response(proc.stderr).text(), proc.exited])
+ expect(exit, stderr).toBe(0)
+ const report = stderr.match(/GITLAB_SDK_LOADED=(true|false)/)
+ expect(report, stderr).not.toBeNull()
+ return report![1] === "true"
+ }
+
+ test("is skipped when GitLab has no credentials", async () => {
+ expect(await importsGitLabSDK({})).toBe(false)
+ }, 30_000)
+
+ test("happens when GitLab has a token", async () => {
+ expect(await importsGitLabSDK({ GITLAB_TOKEN: "test-gitlab-token" })).toBe(true)
+ }, 30_000)
+})
diff --git a/packages/cli/test/provider/provider.test.ts b/packages/cli/test/provider/provider.test.ts
index 70eda32429..e0648a1091 100644
--- a/packages/cli/test/provider/provider.test.ts
+++ b/packages/cli/test/provider/provider.test.ts
@@ -1104,6 +1104,62 @@ test("defaultModel keeps preferring Claude Sonnet 4 for Anthropic", async () =>
})
})
+test("list keeps catalog order, then config-only providers", async () => {
+ await using tmp = await tmpdir({
+ init: async (dir) => {
+ await Bun.write(
+ path.join(dir, "aictrl.json"),
+ JSON.stringify({
+ $schema: "https://aictrl.ai/config.json",
+ provider: {
+ "config-only": {
+ npm: "@ai-sdk/openai-compatible",
+ api: "https://example.test/v1",
+ env: ["CONFIG_ONLY_API_KEY"],
+ models: { "config-model": { name: "Config model" } },
+ },
+ },
+ }),
+ )
+ },
+ })
+ await Instance.provide({
+ directory: tmp.path,
+ init: async () => {
+ Env.set("OPENAI_API_KEY", "test-api-key")
+ Env.set("ANTHROPIC_API_KEY", "test-api-key")
+ Env.set("CONFIG_ONLY_API_KEY", "test-api-key")
+ },
+ fn: async () => {
+ const loaded = Object.keys(await Provider.list())
+ const expected = ["anthropic", "openai", "config-only"]
+ expect(loaded.filter((providerID) => expected.includes(providerID))).toEqual(expected)
+ },
+ })
+})
+
+test("defaultModel takes the first loaded provider in catalog order", async () => {
+ // Other tests leave AWS credentials in process.env, so allow only the two providers under test.
+ await using tmp = await tmpdir({
+ init: async (dir) => {
+ await Bun.write(
+ path.join(dir, "aictrl.json"),
+ JSON.stringify({ $schema: "https://aictrl.ai/config.json", enabled_providers: ["openai", "anthropic"] }),
+ )
+ },
+ })
+ await Instance.provide({
+ directory: tmp.path,
+ init: async () => {
+ Env.set("OPENAI_API_KEY", "test-api-key")
+ Env.set("ANTHROPIC_API_KEY", "test-api-key")
+ },
+ fn: async () => {
+ expect(await Provider.defaultModel()).toEqual({ providerID: "anthropic", modelID: "claude-sonnet-4-6" })
+ },
+ })
+})
+
test("multiple providers can be configured simultaneously", async () => {
await using tmp = await tmpdir({
init: async (dir) => {Metadata
Metadata
Assignees
Labels
P4Backlog - Very lowBacklog - Very lowenhancementNew feature or requestNew feature or requestperformancePerformance optimizationPerformance optimization
Activity