Package: robotstxtr 0.3.1.9000
robotstxtr: Engine-Aware 'robots.txt' Policy with Google's Matcher Algorithm
Evaluates whether URLs may be crawled according to 'robots.txt' rules. A versioned engine-aware facade keeps neutral fetch evidence, per-engine status policy, robots product tokens, and matcher capabilities separate, returning explicit non-decisions when policy is undocumented or a matcher is unavailable. The available matcher is a bundled, upstream-test-validated snapshot of Google's open-source C++ 'robots.txt' matcher, preserving 'Googlebot' behavior. Fetches follow a deterministic, conservative policy, with fetch origins constructed by 'rurl' and every request made through the 'ssrfr' server-side request forgery guard. All network access in tests, examples, and vignettes is mocked, served locally, or refused before any request, so no internet connection is required to install, test, or check the package. This matcher derives from Google's 'robotstxt' project <https://github.com/google/robotstxt>, distributed under the Apache License 2.0; see the package NOTICE.
Authors:
robotstxtr_0.3.1.9000.tar.gz
robotstxtr_0.3.1.9000.zip(r-4.7-x86_64)robotstxtr_0.3.1.9000.zip(r-4.7-arm64)robotstxtr_0.3.1.9000.zip(r-4.6-x86_64)robotstxtr_0.3.1.9000.zip(r-4.6-arm64)robotstxtr_0.3.1.9000.zip(r-4.5-x86_64)
robotstxtr_0.3.1.9000.tgz(r-4.6-x86_64)robotstxtr_0.3.1.9000.tgz(r-4.6-arm64)robotstxtr_0.3.1.9000.tgz(r-4.5-x86_64)robotstxtr_0.3.1.9000.tgz(r-4.5-arm64)
robotstxtr_0.3.1.9000.tar.gz(r-4.7-arm64)robotstxtr_0.3.1.9000.tar.gz(r-4.7-x86_64)robotstxtr_0.3.1.9000.tar.gz(r-4.6-arm64)robotstxtr_0.3.1.9000.tar.gz(r-4.6-x86_64)
robotstxtr_0.3.1.9000.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
robotstxtr/json (API)
| # Install 'robotstxtr' in R: |
| install.packages('robotstxtr', repos = c('https://bart-turczynski.r-universe.dev', 'https://cloud.r-project.org')) |
Bug tracker:https://gitlab.com/bart-turczynski/robotstxtr
Pkgdown/docs site:https://bart-turczynski.gitlab.io
robots-txtrobots-exclusion-protocolweb-crawlercrawl-policygooglebotbingbotyandexurl-matchingrobots-txt-validationcpp
Last updated from:e35323d7a0. Checks:13 ERROR, 2 OK. Indexed: yes.
| Target | Result | Time | Files | Syslog |
|---|---|---|---|---|
| linux-devel-arm64 | ERROR | 280 | ||
| linux-devel-x86_64 | ERROR | 419 | ||
| source / vignettes | OK | 214 | ||
| linux-release-arm64 | ERROR | 413 | ||
| linux-release-x86_64 | ERROR | 477 | ||
| macos-release-arm64 | ERROR | 468 | ||
| macos-release-x86_64 | ERROR | 1484 | ||
| macos-oldrel-arm64 | ERROR | 443 | ||
| macos-oldrel-x86_64 | ERROR | 976 | ||
| windows-devel-arm64 | ERROR | 797 | ||
| windows-devel-x86_64 | ERROR | 510 | ||
| windows-release-arm64 | ERROR | 803 | ||
| windows-release-x86_64 | ERROR | 637 | ||
| windows-oldrel-x86_64 | ERROR | 695 | ||
| wasm-release | OK | 120 |
Exports:allowed_by_robots_textallowed_by_robots_urlas_legacy_robots_decisions_v1robots_bodyrobots_engine_contract_v1robots_engine_contract_v2robots_evaluate_text_v1robots_evaluate_url_v1robots_fetchrobots_resolve_matcher_profile_v1robots_validate_textrobots_validate_url
Dependencies:clicpp11curldigestgluelifecyclepslrpunycoderraddrRcpprlangrurlssrfrstringivctrs
Last update: 2026-11-18
Started: 2026-07-10
Last update: 2026-11-12
Started: 2026-10-28
