[Feature] AI-First Object Storage: Native MCP Server, Agent Auth & Ephemeral Scratch Buckets #7

Summary

Transform MiddleOut into a first-class storage layer for AI agents, LLM bots, and autonomous workflows. This includes shipping a native Model Context Protocol (MCP) server, streamlined Agent Token Authentication (bypassing complex AWS SigV4 signing), ephemeral scratch buckets with TTLs, and llms.txt / OpenAPI tool manifests for automatic tool discovery.

Problem Statement & Motivation

• Agents Need Scratchpad & Artifact Storage: Autonomous agents (Antigravity, Claude Code, Cursor, AutoGen, CrewAI, LangChain) generate images, code dumps, scraped datasets, and intermediate execution state that shouldn't pollute local git repositories.
• SigV4 is Hostile to Simple AI Tool Calls: Generating AWS SigV4 cryptographic signatures inside LLM tool calling is brittle and prone to token exhaustion or signing errors. Agents need clean, token-authenticated REST/JSON endpoints.
• Missing Protocol Standard (MCP): AI tools are converging on Anthropic's Model Context Protocol (MCP). Without an MCP server, bots cannot natively inspect or use MiddleOut storage.

Key Capabilities

1. Native Model Context Protocol (MCP) Server

Provide an MCP server (both stdio and SSE/HTTP transport) bundled directly with MiddleOut:

• middleout-mcp exposing tool definitions:
  • storage_list_buckets(): List accessible buckets and quotas.
  • storage_list_files(bucket, prefix): Browse bucket contents.
  • storage_read_file(bucket, key): Read file contents (auto-detects text vs binary).
  • storage_write_file(bucket, key, content, content_type): Write text/markdown/code directly.
  • storage_upload_artifact(name, content, ttl_hours): Quick scratchpad upload returning an instant shareable URL.
  • storage_get_share_link(bucket, key, expires_in): Generate public or presigned access links.

2. Agent-First Lightweight REST API & Token Auth

In addition to SigV4 and Authorization: MO <key>:<secret>, add a streamlined JSON/REST API designed for bot tool calling:

• Header: Authorization: Bearer mo_agent_<token>
• Endpoints:
  • POST /api/agent/v1/upload (multipart or JSON base64 payload; returns key, size, and instant public/presigned link)
  • GET /api/agent/v1/buckets/:bucket/objects/:key (raw content or JSON with metadata)
  • GET /api/agent/v1/search?q=query (find objects across agent buckets)

3. Ephemeral Scratch Buckets with Auto-Expiry (TTL)

• Allow agents to create temporary scratch buckets for benchmark runs or single sessions:
  • POST /api/agent/v1/scratch-buckets { "name": "run-4821", "ttl": "2h" }
  • Background worker cleans up expired scratch buckets and associated objects automatically.

4. Machine Discovery (llms.txt & OpenAPI Tool Manifest)

• Host /.well-known/llms.txt explaining MiddleOut's capabilities, endpoints, and authentication in LLM-optimized markdown.
• Serve /api/agent/v1/openapi.json matching the OpenAPI 3.1 tool-calling format (compatible with GPT actions, LangChain, and LlamaIndex).

5. Console "Agent Access" Management

• Dedicated Agent Keys section in the MiddleOut Console:
  • Issue scoped agent tokens (restrictable to specific scratch prefixes or buckets).
  • One-click copy for claude_desktop_config.json and Antigravity MCP config snippets.
  • View real-time activity log filtered by agent interactions.

Example MCP Configuration Snippet

Users should be able to drop this into their AI client config:

{
  "mcpServers": {
    "middleout": {
      "command": "npx",
      "args": ["-y", "@middleout/mcp-server"],
      "env": {
        "MIDDLEOUT_ENDPOINT": "http://localhost:8080",
        "MIDDLEOUT_AGENT_KEY": "mo_agent_live_xxxxxxxxxxxx"
      }
    }
  }
}

Implementation Checklist

• Agent Auth & Tokens: Create agent_tokens table with scoping (read, write, admin), TTLs, and rate limits.
• REST Tool Endpoints: Implement /api/agent/v1/* routes returning clean JSON responses.
• MCP Server Package: Build packages/mcp-server (or src/mcp/) using @modelcontextprotocol/sdk.
• TTL & Garbage Collection: Add a lightweight cron/worker to purge expired ephemeral scratch objects.
• Machine Discovery Docs: Add /.well-known/llms.txt and OpenAPI specification generator.
• Console UI: Add "AI & Agents" management tab in the console with copy-ready MCP configs.