Introduction to robotstxtr

What robotstxtr does

robotstxtr decides whether a URL may be crawled according to a site’s robots.txt rules. It answers one question — “is this URL allowed for this crawler?” — and answers it exactly the way Google’s own crawler would, because the matching engine is a vendored, upstream-test-validated snapshot of Google’s open-source C++ robots.txt parser and matcher.

The core value is fidelity. robots.txt matching has many quirks — longest rule wins, Allow beats Disallow on an equal-length tie, * and $ wildcards, case-sensitive paths but case-insensitive user-agent groups, percent-escape canonicalization. Re-implementing those rules by hand is a common source of subtle bugs. robotstxtr sidesteps that by running the real Google algorithm and returning a stable, machine-readable decision for every URL.

This is a faithful, upstream-test-validated replica of the pinned open-source matcher. It is not a claim of universal behavioral identity with Googlebot, which has crawl policy beyond this matcher.

Crawl-decision entry points

There are two ways to get a decision, depending on whether you already have the robots.txt text:

• allowed_by_robots_text() — you supply the robots.txt body yourself. No network access is performed. This is the pure matching path.
• allowed_by_robots_url() — you supply only the target URL(s). The package works out the governing robots.txt origin, fetches it under a deterministic and conservative HTTP policy, and matches.

Both return an S3 robots_decisions object with one row per input URL.

Document validation is a separate operation. robots_validate_text() checks a supplied document, while robots_validate_url() runs the same deterministic fetch stage as robots_fetch() and validates each distinct stored raw body.

Matching against a supplied body

allowed_by_robots_text() needs a robots.txt body, one or more URLs, and a crawler user-agent token. Everything below runs offline, so it is safe to evaluate in this vignette.

robots <- "user-agent: *
disallow: /private
allow: /private/public
"

allowed_by_robots_text(robots, "https://example.com/private/report", "mybot")
#> <robots_decisions>: 1 result, 1 source
#>   allowed: 0  disallowed: 1  unknown: 0
#>  input_id                                url user_agent allowed decision_source
#>         1 https://example.com/private/report      mybot   FALSE   rule_disallow

The result is a data-frame-backed object. Pull out the pieces you need from its results table:

decision <- allowed_by_robots_text(
  robots,
  c(
    "https://example.com/index.html",     # not disallowed, allowed
    "https://example.com/private/report", # under /private, disallowed
    "https://example.com/private/public"  # re-allowed by the Allow rule
  ),
  "mybot"
)

decision$results[c("url", "allowed", "decision_source")]
#>                                  url allowed decision_source
#> 1     https://example.com/index.html    TRUE   default_allow
#> 2 https://example.com/private/report   FALSE   rule_disallow
#> 3 https://example.com/private/public    TRUE      rule_allow

allowed is the headline TRUE/FALSE/NA answer. decision_source explains why: rule_disallow, rule_allow, default_allow (matching ran and nothing matched), missing_allow (the file was absent), or one of the *_unknown values when no safe decision was possible.

allowed_by_robots_text(
  "user-agent: newsbot\ndisallow: /\nuser-agent: *\nallow: /",
  c("https://example.com/a", "https://example.com/a"),
  c("newsbot", "otherbot")
)$results[c("url", "user_agent", "allowed")]
#>                     url user_agent allowed
#> 1 https://example.com/a    newsbot   FALSE
#> 2 https://example.com/a   otherbot    TRUE

m <- allowed_by_robots_text(
  "user-agent: *\ndisallow: /*.pdf$",
  "https://example.com/manual.pdf",
  "mybot"
)
cols <- c("allowed", "matched_line", "matched_rule_type", "matched_rule_value")
m$results[cols]
#>   allowed matched_line matched_rule_type matched_rule_value
#> 1   FALSE            2          disallow            /*.pdf$

robots_body(decision, n = 40)
#> [1] "user-agent: *\ndisallow: /private\nallow: "

Fetching robots.txt for a URL

When you do not already have the robots.txt text, allowed_by_robots_url() does the whole job: it derives the scheme://host[:port]/robots.txt origin for each URL (handling IDN hosts, ports, IPv6 literals, and userinfo stripping via rurl), fetches each distinct origin exactly once under a conservative httr2 policy (10-second timeout, at most five redirects, no HTTPS-to-HTTP downgrades, no cookies or credentials forwarded), and matches every URL against the body it retrieved.

# Not evaluated here: this call performs a network request, which vignettes
# must never require. Run it in an interactive session against a real site.
allowed_by_robots_url("https://example.com/some/page", "mybot")

The returned object has exactly the same robots_decisions shape as allowed_by_robots_text(), with additional fetch metadata populated per row: fetch_outcome (for example fetched, missing, redirect_error, timeout), http_status, and robots_url. A 404/410 origin yields missing_allow (TRUE, because an absent file allows everything); any fetch failure yields fetch_unknown (NA), so a transport problem never silently looks like an allow. A URL that structurally resolves to a private, loopback, link-local, or cloud-metadata address is refused before any request as ssrf_blocked, which also yields fetch_unknown (NA).

If you only want the fetch stage without matching, robots_fetch() returns the grouped fetch results on their own.

Validating the document itself

Use document validation when the question is “is this robots.txt file well-formed for the pinned parser profile?”, rather than “may this crawler fetch this URL?”.

validation <- robots_validate_text(
  "user-agent: *\ndisallow: /private\nunknown-field: value\n"
)
validation$documents
#>   source_id source_type robots_url effective_url http_status fetch_outcome
#> 1  supplied    supplied       <NA>          <NA>          NA      supplied
#>   validation_status                  profile byte_size line_count
#> 1           warning google-parser-compatible        54          3
#>   active_line_count recognized_directives unknown_directives malformed_lines
#> 1                 3                     2                  1               0
#>   user_agent_directives allow_directives disallow_directives sitemap_directives
#> 1                     1                0                   1                  0
#>   unsupported_directives encoding has_bom nul_byte_count final_newline
#> 1                      0    utf-8   FALSE              0          TRUE
#>   max_bytes body_truncated limit_exceeded diagnostic_count warning_count
#> 1        NA          FALSE          FALSE                1             1
#>   error_count
#> 1           0
validation$diagnostics
#>   source_id input_id line severity              code     directive
#> 1  supplied       NA    3  warning unknown_directive unknown-field
#>               raw_text                                                 message
#> 1 unknown-field: value Directive is not recognized by this validation profile.

The documents table contains byte size, line and directive counts, encoding, BOM/NUL and final-newline evidence, and a validation status. The diagnostics table has stable line, severity, code, directive, raw-text, and message columns. Raw vectors are accepted so malformed UTF-8 and NUL bytes can be reported without first converting or losing the acquisition bytes.

# Performs live HTTP, so it is not evaluated in the vignette.
robots_validate_url("https://example.com/some/page")

URL validation reuses robots_fetch() and never parses a truncated body. For example, a decoded body over max_bytes is reported as an acquisition limit failure with validation_status = "not_validated".

The validation profile is explicitly google-parser-compatible: it reflects the package’s pinned parser (including accepted typo and missing-colon forms) plus conservative group and byte checks. It is not universal crawler or full RFC 9309 validation. It also does not infer webmaster intent or fetch declared sitemap resources to verify their availability.

The URL you pass is the URL that is matched

robots.txt matching is sensitive to the exact URL bytes: path case, query strings, fragments, and percent escapes all matter. robotstxtr passes your original URL string to the matcher unchanged — it never cleans, decodes, or reserializes it. The upstream matcher expects an appropriately encoded, RFC 3986 %-escaped full URL, so supplying the URL in that form is the caller’s responsibility.

# Path case is significant: /Private is disallowed, /private is not.
allowed_by_robots_text(
  "user-agent: *\ndisallow: /Private",
  c("https://example.com/Private", "https://example.com/private"),
  "mybot"
)$results[c("url", "allowed")]
#>                           url allowed
#> 1 https://example.com/Private   FALSE
#> 2 https://example.com/private    TRUE

Summary

• allowed_by_robots_text() matches against a body you supply, fully offline.
• allowed_by_robots_url() fetches and matches under a deterministic policy.
• robots_validate_text() and robots_validate_url() diagnose document syntax, structure, byte hazards, and acquisition evidence under an explicit profile.
• Both return a stable, vectorized robots_decisions object with allowed, decision_source, fetch metadata, and per-rule match metadata.
• Decisions come from a vendored snapshot of Google’s own matcher, so the quirky parts of robots.txt semantics are handled faithfully.