High-performance Unicode and Punycode encoding/decoding for internationalized domain names (IDNs) in R.
The punycoder package provides fast, standards-based conversion between Unicode and ASCII representations of domain names, across two distinct surfaces:
puny_encode() / puny_decode() — the raw RFC 3492 transform with xn-- A-label framing (RFC 5890/5891) and letter-digit-hyphen checks, not an IDNA normalization API (no Unicode NFC, UTS #46 mapping, or case folding);host_normalize() — mapping a host name to its canonical lowercase ASCII comparison form under a pinned UTS #46 non-transitional profile.host_normalize() is a UTS #46 profile, not IDNA2008 conformance — UTS #46 is compatibility processing and deliberately accepts labels IDNA2008 would reject (e.g. ☕.example → xn--53h.example). See ?host_normalize and normalization_profile_info() for the normative profile and full standards references (RFC 3492/5890/5891/5892/5893, UTS #46, UAX #15/#44, STD 3, RFC 8753).
Normalization runs against vendored Unicode data, and a build ships a set of Unicode versions with one pinned as the default (currently 17.0.0, with 16.0.0 also shipped). unicode_versions() reports what the installed build carries, and host_normalize(x, unicode_version = "16.0.0") selects another one for a single call — which normalization_profile_info() reflects by appending +unicode-<version> to the reported profile token.
Install the released version of punycoder from CRAN with:
install.packages("punycoder")
Or install the development version from r-universe with:
install.packages(
"punycoder",
repos = c("https://bart-turczynski.r-universe.dev", "https://cloud.r-project.org")
)
To build it from source on GitLab instead:
# install.packages("remotes")
remotes::install_gitlab("bart-turczynski/punycoder")
punycoder needs R 4.1.0 or later and Rcpp, which install.packages() pulls
in. A binary install needs nothing else. A source install needs a C++ compiler,
which the standard R build tools for each platform provide. libidn2 2.3.5 or
later and pkg-config are optional: configure uses them for a native backend
when it finds both.
libidn2)punycoder works without extra system libraries. If libidn2 is available at
build time, the package enables a native backend automatically; otherwise it
uses the built-in C++ fallback backend.
To install the recommended optional dependency:
brew install libidn2 pkg-configsudo apt-get install libidn2-0-dev pkg-configsudo dnf install libidn2-devel pkgconf-pkg-configsudo pacman -S libidn2 pkgconfVerify the library is visible before installing punycoder from source:
system("pkg-config --modversion libidn2")
Then install/reinstall punycoder:
remotes::install_gitlab("bart-turczynski/punycoder")
library(punycoder)
# Basic encoding
puny_encode("café.com")
#> [1] "xn--caf-dma.com"
# Check if domain is punycode
is_punycode("xn--example")
#> [1] TRUE
# Validate domains
validate_domain("test.com")
#> Punycoder Domain Validation Results
#> ==================================
#>
#> 1 domain: 1 valid, 0 invalid (strict = TRUE)
#>
#> Domain: test.com
#> Valid: TRUE
punycoder is a library, not a CLI, but a one-line Rscript wrapper covers the
usual shell need: turning an IDN into the ASCII form that dig, nslookup, or
curl will actually accept. No extra install, no bin/ script. The snippets
below are POSIX sh/bash — keep the -e argument in single quotes so the
shell leaves the R code alone.
Unicode host → ASCII, via the UTS #46 canonical form:
Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)), sep = "\n")' münchen.de
#> xn--mnchen-3ya.de
Use puny_encode() instead when you want the raw RFC 3492 transform with no
UTS #46 mapping, and puny_decode() to go back the other way:
Rscript -e 'cat(punycoder::puny_encode(commandArgs(TRUE)), sep = "\n")' münchen.de
#> xn--mnchen-3ya.de
Rscript -e 'cat(punycoder::puny_decode(commandArgs(TRUE)), sep = "\n")' xn--mnchen-3ya.de
#> münchen.de
Feeding a resolver is then just command substitution (drop sep so nothing but
the host is printed):
dig +short "$(Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)))' münchen.de)"
#> 194.246.166.100
nslookup "$(Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)))' münchen.de)"
All three functions are vectorized, so several hosts can go through one R
startup and out to xargs:
Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)), sep = "\n")' münchen.de москва.рф |
xargs -n 1 dig +short
host_normalize() never aborts: an input it cannot normalize comes back as
NA, which prints as a literal NA line rather than silently disappearing from
the pipeline.
Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)), sep = "\n")' münchen.de 'bad..domain'
#> xn--mnchen-3ya.de
#> NA
puny_encode() and puny_decode() behave differently: they are strict by
default, so a bad input stops the script with a non-zero exit status.
Rscript -e 'cat(punycoder::puny_encode(commandArgs(TRUE)), sep = "\n")' 'bad..domain'
#> Error: Error encoding domain: Domain contains empty label
#> Execution halted
Pass strict = FALSE to get the per-element NA behavior instead, so one bad
host does not take down a batch:
Rscript -e 'cat(punycoder::puny_encode(commandArgs(TRUE), strict = FALSE), sep = "\n")' münchen.de 'bad..domain'
#> xn--mnchen-3ya.de
#> NA
Since NA is printed, not swallowed, drop it before piping to a resolver —
otherwise dig dutifully looks up a host named NA:
Rscript -e 'h <- punycoder::host_normalize(commandArgs(TRUE)); cat(h[!is.na(h)], sep = "\n")' münchen.de 'bad..domain' |
xargs -n 1 dig +short
libidn2 when available, with a built-in fallback backendClean and standardize URL datasets:
# Identify international domains
is_idn(c("café.com", "example.com", "москва.рф"))
# Validate domain names
validate_domain(c("valid.com", "invalid..domain"))
punycoder is a standards primitive for Punycode and host normalization. It is
deliberately agnostic about resolvability and safety; the following are not
part of its acceptance criteria:
host_normalize() is not a safety gate — a successful result says the host is
valid and normalized under the pinned UTS #46 profile, nothing about whether it
is visually safe or non-deceptive. Confusable and restriction-level checks
(UTS #39 / UTR #36, which UTS #46 itself recommends only as application/UI-layer
steps) belong upstack.rurl package, which consumes
punycoder’s host functions.email-to-ASCII helper; splitting an
address and IDNA-encoding its domain part is an addressing concern for an
upstack consumer, not a Punycode primitive.host_normalize()
validates against the pinned UTS #46 profile, not against registry-specific
IDN tables (which evolve independently of Unicode). TLD policy belongs upstack.These opinions belong in higher layers that consume punycoder’s host functions.
punycoder descends from the libidn-based
hrbrmstr/punycode and follows
IDNA2008 / UTS #46 rather than IDNA2003. The
Prior art and comparison
article compares it with
Punycode libraries in other ecosystems and with the comparable R packages, on
the same inputs.
These packages build on data, libraries, and prior work from many others. See ACKNOWLEDGMENTS.md for the full list of thanks.
punycoder is part of a small ecosystem of R packages by the same author:
punycoder for IDNA canonicalization. Use it for eTLD and registrable-domain queries.punycoder and pslr.If you use punycoder in your work, please cite it. Run citation("punycoder")
for the current citation, or see CITATION.cff.
Each release is archived on Zenodo. Cite the concept DOI 10.5281/zenodo.20973629 to refer to the software in general (it always resolves to the latest version), or the version-specific DOI shown on the Zenodo record for a particular release.
We welcome contributions. See CONTRIBUTING.md for the current development workflow, ARCHITECTURE.md for how the package is structured, and DECISIONS.md for the design-decision log.
Please note that this package is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.
MIT