Rule language

The complete WAF-style rule language: rule fields, match conditions, scope-down, rate-based rules, immunity time and custom responses.

Every request AgentGate evaluates gets labels from its signal sources; an ordered rule set turns labels, request facts and rates into an action. For the ideas behind it, start with Rules and managed groups.

Where rule sets come from

• The server's rule set. The built-in default is the managed groups agentgate-core@1, agentgate-agents@1 and agentgate-gateway@1. An operator replaces it with a rules.json file (or RULES_FILE) at start, or at run time with POST /admin/rules on the private admin listener.
• Per-site rules. In the console (site page, Rules) or with PUT /v1/console/sites/{siteKey}/rules a site can store its own rule set, or keep the server's and override single rules' actions. Validate first with POST …/rules/validate. The effective version, <base>+site.r<revision>, is recorded on every decision.

Every audit event and decision lists the labels and the deciding rule.

Rule language reference

A rule set is JSON. Every way of installing one validates it first: every error names the rule, regular expressions are compiled once at that point, and a rule set that fails validation is never installed.

{
  "version": "site-rules-7",
  "default_action": "allow",
  "immunity_seconds": 1800,
  "ip_sets": {"partners": ["198.51.100.0/24", "2001:db8:1::/48"]},
  "rules": [
    {"group": "agentgate-core", "version": 1, "overrides": {"agent_class": "count"}},
    {"group": "agentgate-bot-control", "version": 1},
    {"name": "allow_partners", "action": "allow", "match": {"ip_set": {"name": "partners"}}},
    {"name": "form_flood", "action": "block", "routes": ["/submit"],
     "rate": {"limit": 20, "window_seconds": 60, "keys": ["ip_prefix24"]},
     "response": {"status": 429, "retry_after": 60, "body": {"error": "rate_limited"}}},
    {"group": "agentgate-agents", "version": 1}
  ]
}

Rules are evaluated in order. The first matching rule whose action is not count decides; count rules add agentgate:rule:<name> and continue (every matching rule adds that label). routes limits a rule to evaluation routes: /submit (the form and every browser SDK verification), /agent/submit (the agent API) and /v1/gateway/check (the gateway); omitted means every route.

Rule fields

Match conditions

All conditions present in one match object must hold (AND). Label and score conditions read what sources attached; statements read the request. Statements that need the request (everything except labels, scores and verified_agent) are false when no request data exists.

Nested matches take every condition in this table, to a depth of 8. Limits, enforced at validation: 128 statements per rule, 64 values per text match, regexes 1–512 bytes and 512 per rule set (RE2 syntax, Go regexp: linear time, no backreferences or lookaround), 1,024 inline CIDRs, 64 named sets of up to 10,000 CIDRs. The client address is the connection's, or X-Real-IP from a loopback proxy only (as elsewhere).

scope_down narrows a rule without changing its match; it is most useful on rate-based rules and group references:

{"name": "api_scrapers", "action": "challenge",
 "match": {"any_prefix": ["agentgate:ip:cloud:"]},
 "scope_down": {"path": {"prefix": ["/api/"]}, "method": ["GET"]}}

Rate-based rules

{"name": "per_agent_quota", "action": "block", "routes": ["/agent/submit"],
 "rate": {"limit": 100, "window_seconds": 300, "keys": ["agent"]},
 "response": {"status": 429, "retry_after": 300}}

limit 1–1,000,000 requests per window_seconds (60–600). keys (1–5, combined): ip, ip_prefix24 (/24 for IPv4, /48 for IPv6), asn, session, agent (verified agent or bot name), site (the request's Host header), header:<name>, label:<prefix> (the first label with that prefix, e.g. label:agentgate:tls:ja4:). A rate rule counts every request that reaches it and satisfies its match and scope_down, and matches once that key's count exceeds the limit. match may be empty on a rate rule. A request missing a key component (no ASN data, no verified agent) is not counted and does not match. The window slides (the previous fixed window is weighted by its remaining overlap). Counters are process-local and in memory and kept per site (sites never share a counter): they reset on restart, are not shared between instances, keep at most 20,000 keys per rule (least recently seen evicted) and 1,024 rules; reloading an unchanged rule keeps its counts.

Managed rule group references

A reference {"group": "<name>", "version": N} expands to that group's rules in place. overrides changes single rules' actions (for example to count while you evaluate them) and scope_down is ANDed into every rule of the group. Every group, version and rule is listed on Managed rule groups.

Immunity time

On the session-based form flow (the demo form and /gate.js), after a session passes the visible check, /challenge/verify sets the signed ag_pass cookie for immunity_seconds (60–86,400; default 900, the previous fixed 15 minutes). While it is valid the request carries agentgate:session:passed; the built-in challenge rules exclude that label, and any other rule whose action is challenge is skipped like a count rule (label agentgate:immunity:applied). Blocks still apply. The window is fixed when the pass is issued; lowering immunity_seconds affects new passes only.

Custom responses

"response": {"status": 403, "retry_after": 0,
             "body": {"error": "forbidden", "docs": "https://example.com/bots"},
             "headers": {"Cache-Control": "no-store", "X-Reason": "bot-control"}}

body is a JSON value sent as application/json, or a JSON string sent as text/plain ("body": "go away"); content_type overrides the type. At most 8 KiB, always with X-Content-Type-Options: nosniff. headers allows Cache-Control, Content-Language, Link, Location, Vary, WWW-Authenticate, Access-Control-Allow-Origin, Access-Control-Expose-Headers, Crawler-Price, Crawler-Charged and X-* (not X-Accel-*, X-Sendfile, X-Forwarded-For, X-Real-IP); never Set-Cookie, hop-by-hop or framing headers. Retry-After is the retry_after field. Without body, headers or content_type the response is message as plain text, as before. Challenges on /submit always answer with the challenge JSON; drop always answers like an acceptance.