Getting Started with rurl

Introduction

The rurl package provides tools to parse, normalize, and extract information from URLs using a consistent and safe API. It is fully vectorized and delegates domain handling to the pslr package, which implements the Public Suffix List for accurate domain and TLD extraction.

Safe URL Parsing

Use safe_parse_url() to parse URLs robustly:

safe_parse_url("https://sub.example.co.uk/path?q=1")
#> $original_url
#> [1] "https://sub.example.co.uk/path?q=1"
#> 
#> $scheme
#> [1] "https"
#> 
#> $host
#> [1] "sub.example.co.uk"
#> 
#> $port
#> [1] NA
#> 
#> $path
#> [1] "/path"
#> 
#> $query
#> [1] "q=1"
#> 
#> $fragment
#> [1] NA
#> 
#> $user
#> [1] NA
#> 
#> $password
#> [1] NA
#> 
#> $domain
#> [1] "example.co.uk"
#> 
#> $tld
#> [1] "co.uk"
#> 
#> $domain_ascii
#> [1] "example.co.uk"
#> 
#> $domain_unicode
#> [1] "example.co.uk"
#> 
#> $tld_ascii
#> [1] "co.uk"
#> 
#> $tld_unicode
#> [1] "co.uk"
#> 
#> $is_ip_host
#> [1] FALSE
#> 
#> $clean_url
#> [1] "https://sub.example.co.uk/path"
#> 
#> $parse_status
#> [1] "ok"

The protocol_handling argument controls how schemes are handled:

  • "keep" (default; keeps the current protocol or prepends http:// if missing)
  • "none" (doesn’t add, remove, or change protocols)
  • "strip" (removes protocols)
  • "http" (changes protocols to http:// or adds it if missing)
  • "https" (changes protocols to https:// or adds it if missing)

Extracting URL Components

get_scheme("https://sub.example.com")
#> [1] "https"
get_host("https://sub.example.com")
#> [1] "sub.example.com"
get_path("https://sub.example.com/path/to/page")
#> [1] "/path/to/page"

Each function works on vectors of URLs and gracefully handles NA.

Domain and TLD Parsing

These functions rely on the Public Suffix List:

get_domain("https://a.b.example.co.uk")
#> [1] "example.co.uk"

Extracting TLDs from different sources:

get_tld("https://foo.blogspot.com")
#> [1] "blogspot.com"

Sources include: - "all" (default; will match to the longest available TLD) - "private" (only extract private TLDs) - "icann" (only extract ICANN TLDs)

Vectorization and Edge Cases

All core functions support vectors and handle malformed inputs safely:

urls <- c("example.com", "http://example.com", NA)
get_clean_url(urls)
#> [1] "http://example.com/" "http://example.com/" NA

Advanced Host Manipulation with subdomain_levels_to_keep

Several functions, including safe_parse_url(), get_host(), and get_clean_url(), support the subdomain_levels_to_keep argument. This allows for fine-grained control over how many subdomain levels are preserved in the host component of a URL, after initial www_handling has been applied.

  • NULL (Default): No specific subdomain stripping is performed beyond www_handling.
  • 0: All subdomains are stripped. If www_handling preserved or added ‘www.’, it remains (e.g., ‘www.sub.example.com’ becomes ‘www.example.com’; ‘sub.example.com’ becomes ‘example.com’).
  • N > 0: Keeps up to N levels of subdomains, counted from right-to-left (closest to the registered domain), in addition to any ‘www.’ prefix.

Here are some examples demonstrating its effect on get_host():

get_host(
  "www.three.two.one.example.com",
  subdomain_levels_to_keep = 0
) # www_handling default is "none"
#> [1] "www.example.com"
# Expected: "www.example.com"

get_host(
  "three.two.one.example.com",
  www_handling = "strip",
  subdomain_levels_to_keep = 0
)
#> [1] "example.com"
# Expected: "example.com"

get_host("www.three.two.one.example.com", subdomain_levels_to_keep = 1)
#> [1] "www.one.example.com"
# Expected: "www.one.example.com"

get_host(
  "three.two.one.example.com",
  www_handling = "strip",
  subdomain_levels_to_keep = 1
)
#> [1] "one.example.com"
# Expected: "one.example.com"

get_host(
  "www.three.two.one.example.com",
  www_handling = "keep",
  subdomain_levels_to_keep = 2
)
#> [1] "www.two.one.example.com"
# Expected: "www.two.one.example.com"

And its effect on get_clean_url():

get_clean_url(
  "http://www.deep.sub.example.com/some/path",
  subdomain_levels_to_keep = 0,
  www_handling = "keep"
)
#> [1] "http://www.example.com/some/path"
# yields http://www.example.com/some/path

get_clean_url(
  "http://deep.sub.example.com/some/path",
  subdomain_levels_to_keep = 1
)
#> [1] "http://sub.example.com/some/path"
# yields http://sub.example.com/some/path

Note that get_domain() also accepts subdomain_levels_to_keep, but it does not change the returned domain value. The domain is derived from the host before this specific host modification occurs. The parameter influences the host component that might be used in other parts of the safe_parse_url output, such as the clean_url.

Getting the whole URL back

get_clean_url() is not the only way to turn a parse back into a string, and it is often not the one you want. It is a cleaning product: it drops the fragment and credentials by design and is driven by two dozen presentation dials. Three surfaces exist, and they are deliberately not interchangeable.

u <- "https://user:[email protected]:443/a/../b?q=1#frag"

# (c) cleaning -- an SEO/canonicalization product, intentionally lossy
get_clean_url(u)
#> [1] "https://example.com/a/../b"

# (b) standard serialization -- the full string, exactly as WHATWG would
#     write it. No presentation dial reaches it.
serialize_url(u)
#> [1] "https://user:[email protected]/b?q=1#frag"

# (d) safe display -- for showing a person
format_url(u)
#> [1] "https://<redacted>@example.com:443/b?q=1#frag"

The :443 that survives only in the display output is deliberate: serialize_url() elides a default port because WHATWG’s parse elides it, which is a normalization, and a surface whose job is showing what is really there does not normalize. It shows the port as written.

Use serialize_url() whenever the question is “what does the standard say this URL is”, and format_url() whenever the string is going in front of a human. format_url() redacts credentials, makes invisible and bidirectional-override code points visible as <U+XXXX> tokens, and leaves percent-encoded delimiters encoded so that reading the string cannot suggest structure the URL does not have:

format_url("https://example.com/a%2Fb?x=a%26b%3Dc#%E2%80%AEevil")
#> [1] "https://example.com/a%2Fb?x=a%26b%3Dc#<U+202E>evil"

Its output is display only — never feed it back into a parser, a comparison, or anything that treats it as an address.

Caching

rurl memoizes URL parsing and punycode round-trips to speed repeated operations over large URL vectors; PSL query caching lives in pslr. Inspect, clear, and configure the caches:

rurl_cache_info()                          # entries / enabled / max per cache
rurl_clear_caches()                        # free memory in a long-running session
rurl_cache_config(max_full_parse = 1e4)    # tighten the full-parse bound
rurl_cache_config(puny_encode = FALSE)     # disable a cache entirely

rurl_cache_config() covers three caches: full_parse, puny_encode, and puny_decode. The full_parse cache is bounded at 100000 entries by default: a hard reset-watermark that caps peak memory (once the bound is hit the cache empties completely and rebuilds, rather than evicting per entry). Set max_full_parse = Inf to opt into the historical unbounded behavior. The puny_encode and puny_decode caches are unbounded by design and can be disabled for workloads with very many unique hosts.

Summary

  • Vectorized functions for parsing and cleaning URLs
  • Uses the Public Suffix List for domain logic
  • Unicode/punycode support
  • Three distinct full-string output surfaces: get_clean_url(), serialize_url() and format_url()

See also

rurl is built on two sibling packages that are also available standalone:

  • pslr — Public Suffix List engine. Use it directly for eTLD and registrable-domain queries when you do not need full URL parsing.
  • punycoder — Punycode and IDNA codec for internationalized domain names. Useful for host normalization and Unicode ↔︎ ACE encoding outside the URL context.

Acknowledgments

rurl builds on prior art (notably urltools) and follows published standards throughout — RFC 3986, the WHATWG URL Standard, the Public Suffix List, and UTS #46 for internationalized hosts — while leaning on stringi and the sibling punycoder and pslr packages. URL parsing itself is in-tree.

The full list of credits — prior art, dependencies, the standards this code implements, and the data sources it serves — is in ACKNOWLEDGMENTS.md.