Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .github/workflows/fix-drift.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,27 @@ jobs:
git config user.email "drift-bot@copilotkit.ai"
git checkout -B "fix/drift-$(date +%Y-%m-%d)-${RUN_ID}"

# Provision a local Ollama daemon so the OLLAMA_HOST-gated live drift leg
# (src/__tests__/drift/ollama.drift.ts) is detected here too. Ollama needs
# NO API key — the "credential" is just a running daemon. Pull the smallest
# chat+generate-capable model (qwen2:0.5b, ~350MB); OLLAMA_MODEL points the
# leg at it. COST: install + model pull adds ~2-4 min to the fix run.
- name: Provision Ollama daemon (live drift leg)
run: |
set -euo pipefail
# Download the installer to disk first, then execute it — avoids piping
# a remote, mutable script straight into a shell (no `curl | sh`).
curl -fsSL https://ollama.com/install.sh -o "${RUNNER_TEMP}/ollama-install.sh"
sh "${RUNNER_TEMP}/ollama-install.sh"
ollama serve > /tmp/ollama-serve.log 2>&1 &
for _ in $(seq 1 30); do
if curl -sf http://127.0.0.1:11434/api/version >/dev/null 2>&1; then
echo "ollama daemon ready"; break
fi
sleep 1
done
ollama pull qwen2:0.5b

# Step 1: Detect drift and produce report.
#
# FIX #F3 (round-4) — write the report OUTSIDE the repo checkout, to
Expand All @@ -78,6 +99,9 @@ jobs:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
# Un-gates the Ollama live drift leg against the daemon provisioned above.
OLLAMA_HOST: 127.0.0.1:11434
OLLAMA_MODEL: qwen2:0.5b
PRE_FIX_REPORT: ${{ runner.temp }}/drift-report.json
run: |
set +e
Expand Down Expand Up @@ -210,6 +234,10 @@ jobs:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
# Keep the Ollama live leg un-gated so post-fix verification exercises it
# (the daemon provisioned earlier in this job is still running).
OLLAMA_HOST: 127.0.0.1:11434
OLLAMA_MODEL: qwen2:0.5b
run: pnpm test:drift

# Step 4c: Re-collect drift AUTHORITATIVELY against the LIVE SDK, writing
Expand All @@ -234,6 +262,10 @@ jobs:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
# Keep the Ollama live leg un-gated so the authoritative re-collect
# exercises it (the daemon provisioned earlier in this job is running).
OLLAMA_HOST: 127.0.0.1:11434
OLLAMA_MODEL: qwen2:0.5b
POST_FIX_REPORT: ${{ runner.temp }}/drift-report.post-fix.json
run: |
set +e
Expand Down
25 changes: 25 additions & 0 deletions .github/workflows/test-drift.yml
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,28 @@ jobs:
PREFLIGHT
npx tsx "$PREFLIGHT_FILE"

# Provision a local Ollama daemon so the OLLAMA_HOST-gated live drift leg
# (src/__tests__/drift/ollama.drift.ts) actually runs. Ollama needs NO API
# key — the "credential" is just a running daemon. We pull the smallest
# chat+generate-capable model (qwen2:0.5b, ~350MB) and point the leg at it
# via OLLAMA_MODEL to keep the added cost minimal. COST: the install + model
# pull adds ~2-4 min to every drift run.
- name: Provision Ollama daemon (live drift leg)
run: |
set -euo pipefail
# Download the installer to disk first, then execute it — avoids piping
# a remote, mutable script straight into a shell (no `curl | sh`).
curl -fsSL https://ollama.com/install.sh -o "${RUNNER_TEMP}/ollama-install.sh"
sh "${RUNNER_TEMP}/ollama-install.sh"
ollama serve > /tmp/ollama-serve.log 2>&1 &
for _ in $(seq 1 30); do
if curl -sf http://127.0.0.1:11434/api/version >/dev/null 2>&1; then
echo "ollama daemon ready"; break
fi
sleep 1
done
ollama pull qwen2:0.5b

# Run the collector behind a "retry before alert" wrapper. A single
# critical run can be a transient real-API hiccup (a streaming call
# failing mid-flight), NOT a format change — so the wrapper re-runs the
Expand All @@ -163,6 +185,9 @@ jobs:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
# Un-gates the Ollama live drift leg against the daemon provisioned above.
OLLAMA_HOST: 127.0.0.1:11434
OLLAMA_MODEL: qwen2:0.5b
run: |
set +e
npx tsx scripts/drift-retry.ts
Expand Down
21 changes: 17 additions & 4 deletions src/__tests__/drift/ollama.drift.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,20 @@ import { httpPost, startDriftServer, stopDriftServer } from "./helpers.js";
// Environment-based opt-in (consistent with other drift files)
// ---------------------------------------------------------------------------

const OLLAMA_HOST = process.env.OLLAMA_HOST ?? "http://localhost:11434";
// Accept both a full base URL ("http://127.0.0.1:11434") and Ollama's native
// host:port form ("127.0.0.1:11434" — the value the daemon itself uses). The
// test issues real HTTP requests, so a scheme is required; prefix http:// when
// one is absent.
const RAW_OLLAMA_HOST = process.env.OLLAMA_HOST ?? "http://localhost:11434";
const OLLAMA_HOST = /^https?:\/\//.test(RAW_OLLAMA_HOST)
? RAW_OLLAMA_HOST
: `http://${RAW_OLLAMA_HOST}`;

// The model to exercise against the live daemon. Defaults to "llama3.2" for a
// local developer run, but CI provisions a much smaller model (to keep the
// daemon pull cheap) and points the leg at it via OLLAMA_MODEL. Both /api/chat
// and /api/generate use the same model.
const OLLAMA_MODEL = process.env.OLLAMA_MODEL ?? "llama3.2";

// ---------------------------------------------------------------------------
// Server lifecycle
Expand Down Expand Up @@ -112,7 +125,7 @@ describe.skipIf(!process.env.OLLAMA_HOST)("Ollama drift", () => {
const sdkShape = ollamaChatResponseShape();

const body = {
model: "llama3.2",
model: OLLAMA_MODEL,
messages: [{ role: "user", content: "Say hello" }],
stream: false,
};
Expand Down Expand Up @@ -143,7 +156,7 @@ describe.skipIf(!process.env.OLLAMA_HOST)("Ollama drift", () => {
const sdkChunkShape = ollamaChatStreamChunkShape();

const body = {
model: "llama3.2",
model: OLLAMA_MODEL,
messages: [{ role: "user", content: "Say hello" }],
stream: true,
};
Expand Down Expand Up @@ -181,7 +194,7 @@ describe.skipIf(!process.env.OLLAMA_HOST)("Ollama drift", () => {
const sdkShape = ollamaGenerateResponseShape();

const body = {
model: "llama3.2",
model: OLLAMA_MODEL,
prompt: "Say hello",
stream: false,
};
Expand Down
Loading