Engine-aware contract

library(robotstxtr)

Two axes, selected explicitly

allowed_by_robots_text() and allowed_by_robots_url() always use Google’s matcher. Multi-engine integrations should use the parallel, versioned facade instead, robots_evaluate_text_v1() and robots_evaluate_url_v1(), which make every caller select both axes explicitly:

  • robots_policy_ruleset: how fetch evidence (a missing file, a server error, a redirect) becomes a policy action, per engine.
  • matcher_backend: which matcher evaluates the rules.

The robots_product_token selects the robots user-agent group only; it is never sent as an HTTP header. Everything below runs offline.

robots <- "user-agent: *
disallow: /private
allow: /private/public
"

x <- robots_evaluate_text_v1(
  robots,
  "https://example.com/private/report",
  robots_product_token = "Googlebot",
  robots_policy_ruleset = "google",
  matcher_backend = "google"
)
x
#> <robots_engine_decisions_v1 [2026-08-25.1]>: 1 result, 1 evidence source
#>  input_id                                url robots_policy_ruleset
#>         1 https://example.com/private/report                google
#>  matcher_backend policy_status matcher_status url_decision        reason
#>           google     evaluated      evaluated     disallow rule_disallow

The result keeps neutral fetch evidence, engine policy, matcher availability, and the final URL decision in separate columns:

x$results[c(
  "evidence_status", "policy_status", "matcher_availability",
  "matcher_status", "url_decision", "reason"
)]
#>   evidence_status policy_status matcher_availability matcher_status
#> 1     usable_body     evaluated            available      evaluated
#>   url_decision        reason
#> 1     disallow rule_disallow

Explicit non-decisions

An unavailable matcher returns capability_unavailable rather than silently falling back to Google. The RFC 9309 backend is such a matcher:

robots_evaluate_text_v1(
  robots,
  "https://example.com/private/report",
  robots_product_token = "mybot",
  robots_policy_ruleset = "assumed_rfc9309",
  matcher_backend = "rfc9309"
)$results[c("matcher_status", "reason", "url_decision")]
#>           matcher_status                         reason url_decision
#> 1 capability_unavailable matcher_capability_unavailable         <NA>

Bounded vendor profiles

The Google backend accepts any valid product token and returns Google semantics for it, which is not a claim about the crawler that token names. The Yandex and Bing backends are bounded to their supported profiles and never generalize to other tokens:

contract <- robots_engine_contract_v1()
contract$matcher_availability
#>                   google                   yandex                  rfc9309 
#>              "available"              "available" "capability_unavailable" 
#>                     bing 
#>              "available"
contract$matcher_capability$yandex$supported_profiles[
  c("profile_id", "accepted_token", "profile_revision")
]
#>              profile_id      accepted_token profile_revision
#> 1                yandex              Yandex     yandex-0.1.0
#> 2 yandex_additional_bot YandexAdditionalBot     yandex-0.1.0
contract$matcher_capability$bing$supported_profiles[
  c("profile_id", "accepted_token", "profile_revision")
]
#>   profile_id accepted_token      profile_revision
#> 1    bingbot        bingbot  bingbot-2026-07-23.1
#> 2   adidxbot       adidxbot adidxbot-2026-07-23.1

Any other token on those backends is a checked non-decision, not a decision:

robots_evaluate_text_v1(
  robots,
  "https://example.com/private/report",
  robots_product_token = "YandexImages",
  robots_policy_ruleset = "yandex",
  matcher_backend = "yandex"
)$results[c("matcher_status", "reason", "url_decision")]
#>   matcher_status                    reason url_decision
#> 1  not_evaluated unsupported_product_token         <NA>

The Yandex and Bing backends are independent, unofficial compatibility profiles and make no production-crawler parity claim.

Contract metadata

robots_engine_contract_v1() publishes the schema revision, policy and matcher revisions, and capability metadata; robots_engine_contract_v2() adds the extended matcher-status set that the Bing backend uses.

contract$contract_id
#> [1] "robotstxtr.engine-aware/v1"
contract$schema_revision
#> [1] "2026-08-25.1"
robots_engine_contract_v2()$matcher_status_set
#> [1] "evaluated"                    "not_needed"                  
#> [3] "not_evaluated"                "capability_unavailable"      
#> [5] "invalid_request_target"       "unsupported_profile"         
#> [7] "matcher_input_limit_exceeded" "matcher_work_limit_exceeded"

as_legacy_robots_decisions_v1() converts a v1 result evaluated with the Google policy and the Google matcher to the robots_decisions shape the headline functions return. The v1 contract is specified in the engine-aware contract design document, and the Bing backend and the v2 status set in the Bing integration v2 specification.