Repository navigation
[Feature Proposal] Support for Labeling and Filtering Substrate Entities #186
Description
Currently, the Actor entity in Agent Substrate lacks support for arbitrary key-value metadata (labels/tags). As the platform grows, users and operators need a flexible way to categorize, group, and query active workloads without forcing database-specific indexing implementation requirements.
We propose adding a map<string, string> labels field to the Actor resource, enabling Kubernetes-style label selectors for filtering during ListActors requests.
Note
Performance Characteristic: ListActors is an operational control-plane query (used by command-line tools like kubectl-ate, admin UIs, and operator scripts) and is not on the latency-critical path for routing, scheduling, suspending, or resuming actors. Consequently, in-memory filtering at the service layer is acceptable and does not impact critical path latency.
User Stories & Use Cases
1. Ownership & Multi-Tenancy
- As a developer using a shared Substrate cluster, I want to filter actor listings to only see my own workloads (e.g.
owner=alice). - As a team lead, I want to list all actors billed to my team (e.g.
team=checkout).
2. Environment & Resource Lifecycle Management
- As an operator, I want to run a cleanup script that lists all actors in staging (e.g.
env=staging) and deletes/suspends them at the end of the day, without affecting production workloads.
3. Business & Workflow Context
- As a platform engineer, I want to look up an active agent instance by the customer id it is serving (e.g.
customer_id=12345) to inspect its status or troubleshoot a support ticket.
4. Scheduler Placement constraints (Future proofing)
- As a scheduler developer, I want actors to carry constraints (e.g.
gpu=required,region=us-central) so that the orchestrator can schedule them onto matching worker pods.
Proposed Design
1. Protobuf API Changes
We will update pkg/proto/ateapipb/ateapi.proto to include a labels map in the Actor message and the creation requests, along with a string-based label_selector for query filtering.
message Actor {
...
// Labels associated with the actor.
map<string, string> labels = 12;
}
message CreateActorRequest {
...
// Optional labels to apply to the actor upon creation.
map<string, string> labels = 4;
}
message ListActorsRequest {
...
// An optional Kubernetes-style label selector query string (e.g. "env=production,app=billing").
string label_selector = 3;
}2. Database-Agnostic Filtering
To keep the persistence layer pluggable and simple:
- The Database Interface (
store.Interface) will remain simple and only handle standard paginated key scanning. It will not know about the filter criteria. - The Service Layer (
controlapi.ListActors) will fetch the page of raw actors and apply the filter evaluation in memory using the Gok8s.io/apimachinery/pkg/labelspackage. - Since our Redis storage driver uses
protojsonfor marshaling/unmarshaling, the newlabelsfield will automatically persist to Redis without requiring schema changes or updates to Redis Lua scripts.
Implementation Plan
- Update
pkg/proto/ateapipb/ateapi.protoand re-generate Go protobuf bindings. - Update the persistence layer interface and implementations to support setting and retrieving labels.
- Integrate
k8s.io/apimachinery/pkg/labelsat the service layer incmd/ateapi/internal/controlapi/list_actors.goto filter list results before returning them. - Add unit and integration tests covering label assignment, persistence, and selector querying.
Activity
eliranw commentedon Jun 8, 2026
+1 from me, I’d use this. My case is plain metadata: tag actors with things like owner, app, or env and read it back on Get/List.
One thing I noticed though: the proposal only sets labels at creation, on CreateActorRequest. There’s no way to change them afterward. That’s fine for some of this, but we might also need re-labeling on an already existing actor, maybe for promotion of an actor between envs, or attaching metadata to an existing actor.
The Control service has no general update RPC today; actors only change through suspend, resume, and delete. So mutable labels would need a new RPC and a new write path, not just another field. Creation-only labels are basically free since protojson already persists them.
dims commentedon Jun 8, 2026
Grant McCloskey (@MushuEE)
owner=bobtoo)Also +1 to Mutability from eliranw above.
thockin commentedon Jun 10, 2026
I think this is mostly fine, but I want to ask a few questions before we commit to the shape of it:
Can an ActorTemplate specify default labels? I guess that can be a distinct proposal, including which layer "wins" in case of conflict.
ListActors is something we need to think REALLLLY carefully about. If we have 100M actors and each on is a few hundred bytes, that is going to be tens of gigabytes of data. We HAVE to do some sort of pagination and we need to be careful that selectors are not too complex to evaluate.
We should spec the key and value format tightly. Probably just define it the same as k8s?
If a selector is a string we need to define a minimal grammar -- is it JUST an "and" expression or can we specify "or" and other boolean operators. What about other operators that emerged in k8s (in, notin, exists, ...)? It's OK if we don't solve these now, but we should not just pretend these problems don't exist. And I think defining a grammar is better than defining an AST in JSON. :)
We will need mutation of labels, almost certainly.
This is not a replacement for Design: Should Agent Substrate have its own concept of “namespaces”? #21 (namespaces)
thockin commentedon Jun 10, 2026
FWIW I talked to Joe Betz (@jpbetz) about how he would retool k8s selectors if he could.
His suggestion was to stay a subset of CEL, even if we don't use the CEL evaluator. Something like:
==,!=,x in [a, b, c],!(x in [a, b, c]),||,&&label lookups would be something like
labels["x"] == "value". By default CEL treats a failed map lookup as an error, but we could configure the map to think it has all keys, and just return and "undef" which is not equal to any valueptone commentedon Oct 1, 2026
Here is additional feedback and concrete integration findings from multi-tenant testing on Substrate
d277088bregarding labels, annotations, and filtering:Summary
Actors and ActorTemplates carry no labels, annotations or free-form metadata. Because of this, a client cannot record its own identity for an actor (tenant, owner, agent id) on the actor itself. Atespace and actor names are length-limited, so encoding identity into names is lossy. A client that keeps the mapping in memory loses it on restart, and every existing actor becomes unidentifiable.
Impact
Environment
d277088b(v1alpha), ateapi proto.Steps to reproduce / evidence
d277088b,pkg/proto/ateapipb/ateapi.proto):ResourceMetadata(from line 230) carries atespace, name, uid, version and timestamps.Actor(from line 352) andActorTemplate(from line 752) have no map field.map<string, string>fields in the proto are:Selector.match_labels(line 224);volume_context(line 349);Worker.labels, which mirror WorkerPool labels (line 1912).Proposed behaviour / API sketch
map<string,string> labelsandmap<string,string> annotationstoResourceMetadata, or toActorandActorTemplate, with Kubernetes-style key and value validation. They should be settable at create and returned by Get and List.ListActors(andListActorTemplates), for examplelabel_selector: "tenant=abc,agent=xyz".Workaround
An integrator can:
In the tested integration, roughly 400 lines of record-less actor detection and fail-closed handling exist only because of this gap.
QuentinBisson commentedon Oct 22, 2026
We have a use case for this from the egress side. When an egress policy denies a destination, the agent's owners want to know which agent hit it, but the gateway only sees the actor name, which the creating platform (kagent) sets to a session id.
Labels set at creation, for example an agent name, would let the gateways copy an allowlisted set of label keys onto their denial telemetry. We would not need to change labels after creation, and an ActorTemplate default would cover our case.
QuentinBisson commentedon Oct 26, 2026
Grant McCloskey (@MushuEE) are you still on this? If not, we'd like to start with a small first step to be able to unlock our use case above
QuentinBisson commentedon Oct 26, 2026
Here is a small PR that covers our need #3208
Actors carry labels set at creation, with defaults from their template
Actors carry labels set at creation, with defaults from their template
Actors carry labels set at creation, with defaults from their template
Actors carry labels set at creation, with defaults from their template