| Title: | Fetch Files Through a Real Browser When a Plain Request is Refused |
|---|---|
| Description: | Some servers refuse any request that does not come from a web browser, with a browser check or a block page instead of the file, 'Cloudflare' being the common case. This package opens a real 'Chrome' with a throwaway profile, lets any check run, and fetches the file or page: through 'curl' carrying the clearance the browser was given where the server accepts that, and through the browser itself where it does not. Nothing is simulated; where a check asks a person to click, a window waits for them. Built on 'chromote'. |
| Authors: | Jens von Bergmann [aut, cre] |
| Maintainer: | Jens von Bergmann <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.1.0 |
| Built: | 2026-11-20 13:27:21 UTC |
| Source: | https://github.com/cran/safeconduct |
Can this package open a browser here?
browser_available()browser_available()
TRUE, or FALSE with a reason attribute: the
safeconduct.browser option is FALSE, or no Chrome or Chromium was
found (the CHROMOTE_CHROME environment variable points at one).
browser_available() attr(browser_available(), "reason")browser_available() attr(browser_available(), "reason")
Opens a Chrome with a throwaway profile on the host's root, lets any check
run, and keeps it open for read_page() and fetch() through their
session argument. Close it with close(); otherwise it lives until the
R session ends. A browser that has gone away, its window closed, is
reopened the same way at the next call, which says so.
browser_session( url, quiet = FALSE, wait = NULL, headed = "auto", useragent = TRUE )browser_session( url, quiet = FALSE, wait = NULL, headed = "auto", useragent = TRUE )
url |
A URL on the host. |
quiet |
Suppress progress messages. |
wait |
Seconds to wait for the check to pass: by default two minutes with a window, so that a person can click, and thirty seconds headless. |
headed |
How to run the browser when one is needed. |
useragent |
The browser's user-agent string. |
A safeconduct_session, whose clearance is what clearance()
would return. Errors as fetch()'s.
## Not run: s <- browser_session("https://results.vancouver.ca/") r <- read_page("https://results.vancouver.ca/data/arearesults.json", session = s) close(s) ## End(Not run)## Not run: s <- browser_session("https://results.vancouver.ca/") r <- read_page("https://results.vancouver.ca/data/arearesults.json", session = s) close(s) ## End(Not run)
Opens a Chrome on the host, lets any check run, and returns the browser's
user-agent string and cookies, which clearance_handle() puts on a curl
request. Whether the server then serves curl is the server's call: after
a block page it usually does not, since the rule is about the client, and
fetch() then downloads in the browser instead. Clearances are cached per
host for the R session.
clearance( url, refresh = FALSE, quiet = FALSE, wait = NULL, headed = "auto", useragent = TRUE )clearance( url, refresh = FALSE, quiet = FALSE, wait = NULL, headed = "auto", useragent = TRUE )
url |
The file's URL. |
refresh |
Get a new clearance even when one is cached. |
quiet |
Suppress progress messages. |
wait |
Seconds to wait for the check to pass: by default two minutes with a window, so that a person can click, and thirty seconds headless. |
headed |
How to run the browser when one is needed. |
useragent |
The browser's user-agent string. |
A safeconduct_clearance: a list with host, useragent,
cookie (one Cookie: header value) and time. Errors as fetch()'s.
## Not run: cl <- clearance("https://www12.statcan.gc.ca/") curl::curl_fetch_memory("https://www12.statcan.gc.ca/", clearance_handle(cl)) ## End(Not run)## Not run: cl <- clearance("https://www12.statcan.gc.ca/") curl::curl_fetch_memory("https://www12.statcan.gc.ca/", clearance_handle(cl)) ## End(Not run)
The user-agent string and the cookies go on a request together: Cloudflare ties the clearance cookie to the user agent that earned it, and the cookie alone is answered with the check again.
clearance_handle(clearance = NULL, ...)clearance_handle(clearance = NULL, ...)
clearance |
A clearance from |
... |
Further options for |
A curl handle with followlocation set, and the clearance's
useragent and cookie when one was given.
h <- clearance_handle(list(useragent = "Mozilla/5.0", cookie = "cf_clearance=x"))h <- clearance_handle(list(useragent = "Mozilla/5.0", cookie = "cf_clearance=x"))
Opens a Chrome with a throwaway profile on the host, lets any check run,
and downloads url in it, with no plain request first. fetch() does
this only when it has to.
download_in_browser( url, destfile, quiet = FALSE, timeout = 3600, headed = "auto", useragent = TRUE )download_in_browser( url, destfile, quiet = FALSE, timeout = 3600, headed = "auto", useragent = TRUE )
url |
The file's URL. |
destfile |
Where to write it. A failed transfer leaves nothing there. |
quiet |
Suppress progress messages. |
timeout |
Seconds a transfer in the browser may take. |
headed |
How to run the browser when one is needed. |
useragent |
The browser's user-agent string. |
destfile, invisibly. Errors as fetch()'s.
## Not run: download_in_browser("https://example.org/large.zip", "large.zip") ## End(Not run)## Not run: download_in_browser("https://example.org/large.zip", "large.zip") ## End(Not run)
Fetches url to destfile. A plain request is made first, and if the
file comes no browser is involved. If the server answers with a browser
check (is_challenge()) or a block page (is_block()), a Chrome with a
throwaway profile is opened on the host, any check runs, and the file is
downloaded: through curl carrying the browser's clearance where the server
accepts that, and in the browser itself where it does not. A clearance
curl can use is kept for the rest of the R session, so later files from
the host need no browser.
fetch( url, destfile, quiet = FALSE, timeout = 3600, session = NULL, headed = "auto", useragent = TRUE )fetch( url, destfile, quiet = FALSE, timeout = 3600, session = NULL, headed = "auto", useragent = TRUE )
url |
The file's URL. |
destfile |
Where to write it. A failed transfer leaves nothing there. |
quiet |
Suppress progress messages. |
timeout |
Seconds a transfer in the browser may take. |
session |
A |
headed |
How to run the browser when one is needed. |
useragent |
The browser's user-agent string. |
Nothing is simulated. A check that shows a "Verify you are human" box
waits in a window, for up to two minutes, for a person to click it. The
window is brought to the front when a check shows in it, and wants to be
left there until it is through: a check run with the window behind
something else is reissued on every click. A
file downloaded in the browser lands under tempdir() and is moved to
destfile when complete.
destfile, invisibly. Errors are classed under
safeconduct_error: safeconduct_unavailable when no browser can be
opened here, safeconduct_not_passed when the check was not passed in
time or the server blocks the browser too, with a reason field, and
safeconduct_download_error when the transfer failed.
Chrome or Chromium, found by chromote::find_chrome(); the
CHROMOTE_CHROME environment variable points at one somewhere unusual,
and browser_available() says whether one can be opened here.
options(safeconduct.browser = FALSE) never opens a browser; the window's
size and the other options are described in safeconduct-package.
At the console, once a host has been found to need a window or the browser
for the transfer, a tip says how to skip the finding out next time:
headed = TRUE, or a browser_session(). Tips come once per host and R
session, not when quiet, and not for calls made from another package.
They are messages of class safeconduct_hint.
## Not run: url <- paste0("https://www12.statcan.gc.ca/census-recensement/2021/", "geo/sip-pis/rnf-frr/files-fichiers/lrnf000r21a_e.zip") fetch(url, "lrnf000r21a_e.zip") ## End(Not run)## Not run: url <- paste0("https://www12.statcan.gc.ca/census-recensement/2021/", "geo/sip-pis/rnf-frr/files-fichiers/lrnf000r21a_e.zip") fetch(url, "lrnf000r21a_e.zip") ## End(Not run)
Cloudflare's firewall can refuse a client that does not look like a
browser outright, with a 403 and its own block page, while a real
browser is served.
is_block(status, headers, body = NULL)is_block(status, headers, body = NULL)
status |
The HTTP status code. |
headers |
The response headers, as the raw header block (one string,
lines separated by |
body |
The response body, as a string. |
TRUE for that page: a 403 from server: cloudflare whose body
says "Sorry, you have been blocked".
is_block(403, "HTTP/2 403\r\nserver: cloudflare\r\n\r\n", "<html><title>Attention Required! | Cloudflare</title>...") is_block(403, "HTTP/2 403\r\nserver: nginx\r\n\r\n", "Forbidden")is_block(403, "HTTP/2 403\r\nserver: cloudflare\r\n\r\n", "<html><title>Attention Required! | Cloudflare</title>...") is_block(403, "HTTP/2 403\r\nserver: nginx\r\n\r\n", "Forbidden")
Does a response carry a browser check instead of the resource?
is_challenge(headers)is_challenge(headers)
headers |
The response headers, as the raw header block (one string,
lines separated by |
TRUE for a check a real browser would pass: Cloudflare's, by its
cf-mitigated: challenge header.
is_block() for the other refusal a browser gets past.
is_challenge("HTTP/2 403\r\ncf-mitigated: challenge\r\n\r\n") is_challenge("HTTP/2 200\r\ncontent-type: application/zip\r\n\r\n")is_challenge("HTTP/2 403\r\ncf-mitigated: challenge\r\n\r\n") is_challenge("HTTP/2 200\r\ncontent-type: application/zip\r\n\r\n")
page_text() decodes the body to a UTF-8 string, page_json() parses it
with jsonlite::fromJSON(), and page_html() with xml2::read_html().
page_text(resp, encoding = NULL, check = TRUE) page_json(resp, ..., check = TRUE) page_html(resp, encoding = NULL, check = TRUE)page_text(resp, encoding = NULL, check = TRUE) page_json(resp, ..., check = TRUE) page_html(resp, encoding = NULL, check = TRUE)
resp |
A response from |
encoding |
The body's charset, overriding the response's. |
check |
Whether an HTTP status outside 200–299 is an error. |
... |
Passed on to |
The charset comes from the response's content-type; without one,
page_text() and page_json() assume UTF-8 and page_html() leaves it
to the page's own <meta charset>. ISO-8859-1 is read as windows-1252, as
browsers do. A body that is not valid in its charset is an error. A "not
found" page served with status 200 passes the status check; page_json()
refuses an HTML body, and for a page, compare resp$url with the URL
asked for.
page_text(): a UTF-8 string. page_json(): what
jsonlite::fromJSON() returns. page_html(): an xml_document whose
relative links resolve against resp$url. Errors are classed under
safeconduct_error: safeconduct_http_error for a status outside
200–299, carrying status and url, and safeconduct_parse_error for
a body that cannot be decoded, or HTML where JSON was asked for.
## Not run: page_json("https://results.vancouver.ca/data/arearesults.json") page_html("https://www12.statcan.gc.ca/census-recensement/index-eng.cfm") ## End(Not run)## Not run: page_json("https://results.vancouver.ca/data/arearesults.json") page_html("https://www12.statcan.gc.ca/census-recensement/index-eng.cfm") ## End(Not run)
Sends a GET for the first byte, so that the answer carries the
resource's headers without the resource; a HEAD is not used because
some servers answer it differently.
probe(url, clearance = NULL)probe(url, clearance = NULL)
url |
The URL. |
clearance |
A clearance from |
A list: reachable, status (NA when unreachable),
challenged (is_challenge()), blocked (is_block()), type (the
content-type, lower case), size (bytes, from content-range or
content-length, NA when unknown) and headers (a named list). A
browser is called for when challenged or blocked is TRUE.
## Not run: probe("https://www12.statcan.gc.ca/census-recensement/index-eng.cfm") ## End(Not run)## Not run: probe("https://www12.statcan.gc.ca/census-recensement/index-eng.cfm") ## End(Not run)
fetch() for a page or a feed: the response is returned rather than
written to disk, shaped like curl::curl_fetch_memory()'s, by the same
route; where only the browser is served, the page the browser holds on
the host reads it. For a feed read every few minutes, pass a
browser_session() instead of opening a browser per call; a check the
host asks for again meanwhile is passed and the read repeated.
read_page( url, session = NULL, quiet = FALSE, timeout = 120, headed = "auto", useragent = TRUE )read_page( url, session = NULL, quiet = FALSE, timeout = 120, headed = "auto", useragent = TRUE )
url |
The file's URL. |
session |
A |
quiet |
Suppress progress messages. |
timeout |
Seconds a read in the browser may take. |
headed |
How to run the browser when one is needed. |
useragent |
The browser's user-agent string. |
A list: url (after redirects), status_code, type (the
content-type, lower case), headers (a named list) and content
(raw). An HTTP error status is returned, not raised. Parse it with
page_text(), page_json() or page_html(). Errors as fetch()'s.
## Not run: r <- read_page("https://results.vancouver.ca/data/arearesults.json") page_json(r) ## End(Not run)## Not run: r <- read_page("https://results.vancouver.ca/data/arearesults.json") page_json(r) ## End(Not run)