Package: robotstxtr 0.3.1.9000

robotstxtr: Engine-Aware 'robots.txt' Policy with Google's Matcher Algorithm

Evaluates whether URLs may be crawled according to 'robots.txt' rules. A versioned engine-aware facade keeps neutral fetch evidence, per-engine status policy, robots product tokens, and matcher capabilities separate, returning explicit non-decisions when policy is undocumented or a matcher is unavailable. The available matcher is a bundled, upstream-test-validated snapshot of Google's open-source C++ 'robots.txt' matcher, preserving 'Googlebot' behavior. Fetches follow a deterministic, conservative policy, with fetch origins constructed by 'rurl' and every request made through the 'ssrfr' server-side request forgery guard. All network access in tests, examples, and vignettes is mocked, served locally, or refused before any request, so no internet connection is required to install, test, or check the package. This matcher derives from Google's 'robotstxt' project <https://github.com/google/robotstxt>, distributed under the Apache License 2.0; see the package NOTICE.

Authors:Bart Turczynski [aut, cre, cph], Google LLC [cph]

robotstxtr_0.3.1.9000.tar.gz
robotstxtr_0.3.1.9000.zip(r-4.7-x86_64)robotstxtr_0.3.1.9000.zip(r-4.7-arm64)robotstxtr_0.3.1.9000.zip(r-4.6-x86_64)robotstxtr_0.3.1.9000.zip(r-4.6-arm64)robotstxtr_0.3.1.9000.zip(r-4.5-x86_64)
robotstxtr_0.3.1.9000.tgz(r-4.6-x86_64)robotstxtr_0.3.1.9000.tgz(r-4.6-arm64)robotstxtr_0.3.1.9000.tgz(r-4.5-x86_64)robotstxtr_0.3.1.9000.tgz(r-4.5-arm64)
robotstxtr_0.3.1.9000.tar.gz(r-4.7-arm64)robotstxtr_0.3.1.9000.tar.gz(r-4.7-x86_64)robotstxtr_0.3.1.9000.tar.gz(r-4.6-arm64)robotstxtr_0.3.1.9000.tar.gz(r-4.6-x86_64)
robotstxtr_0.3.1.9000.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
robotstxtr/json (API)

# Install 'robotstxtr' in R:
install.packages('robotstxtr', repos = c('https://bart-turczynski.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://gitlab.com/bart-turczynski/robotstxtr

Pkgdown/docs site:https://bart-turczynski.gitlab.io

Uses libs:
  • c++– GNU Standard C++ Library v3

On CRAN:

Conda:

robots-txtrobots-exclusion-protocolweb-crawlercrawl-policygooglebotbingbotyandexurl-matchingrobots-txt-validationcpp

5.65 score 1 packages 577 downloads 12 exports 15 dependencies

Last updated from:e35323d7a0. Checks:13 ERROR, 2 OK. Indexed: yes.

TargetResultTimeFilesSyslog
linux-devel-arm64ERROR280
linux-devel-x86_64ERROR419
source / vignettesOK214
linux-release-arm64ERROR413
linux-release-x86_64ERROR477
macos-release-arm64ERROR468
macos-release-x86_64ERROR1484
macos-oldrel-arm64ERROR443
macos-oldrel-x86_64ERROR976
windows-devel-arm64ERROR797
windows-devel-x86_64ERROR510
windows-release-arm64ERROR803
windows-release-x86_64ERROR637
windows-oldrel-x86_64ERROR695
wasm-releaseOK120

Exports:allowed_by_robots_textallowed_by_robots_urlas_legacy_robots_decisions_v1robots_bodyrobots_engine_contract_v1robots_engine_contract_v2robots_evaluate_text_v1robots_evaluate_url_v1robots_fetchrobots_resolve_matcher_profile_v1robots_validate_textrobots_validate_url

Dependencies:clicpp11curldigestgluelifecyclepslrpunycoderraddrRcpprlangrurlssrfrstringivctrs

Introduction to robotstxtr
What robotstxtr does | Crawl-decision entry points | Matching against a supplied body | Vectorized, with per-URL user agents | Rich match metadata | Inspecting the stored body | Fetching robots.txt for a URL | Validating the document itself | The URL you pass is the URL that is matched | Summary

Last update: 2026-11-18
Started: 2026-07-10

Engine-aware contract
Two axes, selected explicitly | Explicit non-decisions | Bounded vendor profiles | Contract metadata

Last update: 2026-11-12
Started: 2026-10-28