1. Crawl and collect text | 2. Chunk | 3. Embed | 4. Export for retrieval | End to end
This package is considered a duplicate. The official version of this package is found at:https://strategicprojects.r-universe.dev/crawlee
Package: crawlee 0.1.0
Andre Leite
crawlee: Tidy Interface for Reproducible Web Crawling
A tidy, pipe-friendly toolkit for reproducible web crawling and structured data collection, inspired by the architecture of the 'Crawlee' library. Provides a unified crawler with a deduplicating, resumable request queue, content-type aware handlers, structured storage backends and rich console logging via 'cli'. Supports crawling HTML pages, sitemaps, RSS and Atom feeds and PDF documents, with optional headless-browser rendering and helpers for retrieval-augmented generation.
Authors:
crawlee_0.1.0.tar.gz
crawlee_0.1.0.tar.gz(r-4.7-any)crawlee_0.1.0.tar.gz(r-4.6-any)
crawlee_0.1.0.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
crawlee/json (API)
| # Install 'crawlee' in R: |
| install.packages('crawlee', repos = c('https://cran.r-universe.dev', 'https://cloud.r-project.org')) |
Bug tracker:https://github.com/strategicprojects/crawlee/issues
Pkgdown/docs site:https://strategicprojects.github.io
Last updated from:79ad61db59. Checks:4 OK. Indexed: no.
| Target | Result | Time | Files | Syslog |
|---|---|---|---|---|
| linux-devel-x86_64 | OK | 156 | ||
| source / vignettes | OK | 243 | ||
| linux-release-x86_64 | OK | 151 | ||
| wasm-release | OK | 163 |
Exports:cr_autoscalecr_chunkcr_closecr_collectcr_datasetcr_embedcr_exportcr_from_rsscr_from_sitemapcr_normalize_urlcr_on_htmlcr_on_pdfcr_optionscr_parallelcr_persistcr_runcr_storecr_streamcr_use_browsercr_use_httpcrawlerCrawlerDatasetKeyValueStoreRequestQueue
Dependencies:askpassclicurlgluehttrhttr2jsonlitelifecyclemagrittrmimeopensslpillarpkgconfigR6rappdirsrlangrvestselectrsystibbleutf8vctrswithrxml2
A RAG pipeline
Last update: 2026-07-03
Started: 2026-07-03
Crawling a website
The model | Your first crawler | Following links | Controlling scope | Routing different page types | Crawling from a sitemap | Rendering JavaScript pages | Where next
Last update: 2026-07-03
Started: 2026-07-03
Getting started with crawlee
The mental model | A minimal crawl | The handler context | Controlling link discovery | Reproducibility
Last update: 2026-07-03
Started: 2026-07-03
Scaling and politeness
Being a good web citizen | Going faster | Fixed-concurrency batches — cr_parallel() | Adaptive batches — cr_autoscale() | Continuous streaming pool — cr_stream() | Choosing an engine | Combining with persistence
Last update: 2026-07-03
Started: 2026-07-03
Storage and resumable runs
The dataset | The key-value store | The request queue and reproducibility | One-call setup: cr_persist() | Resuming
Last update: 2026-07-03
Started: 2026-07-03
Readme and manuals
Help Manual
| Help page | Topics |
|---|---|
| Enable autoscaled parallel fetching | cr_autoscale |
| Chunk text for retrieval-augmented generation | cr_chunk |
| Release a crawler's resources | cr_close |
| Collect crawl results | cr_collect |
| Configure the dataset backend | cr_dataset |
| Attach embeddings to chunks | cr_embed |
| Export chunks (and embeddings) for retrieval | cr_export |
| Discover URLs from an RSS or Atom feed | cr_from_rss |
| Discover URLs from a sitemap | cr_from_sitemap |
| Normalise a URL into a canonical form | cr_normalize_url |
| Register an HTML handler | cr_on_html |
| Register a PDF handler | cr_on_pdf |
| Set crawler options | cr_options |
| Enable parallel (concurrent) fetching | cr_parallel |
| Persist a crawl to a run directory (and resume it) | cr_persist |
| Run a crawl | cr_run |
| Configure the key-value store for binary content | cr_store |
| Enable the streaming scheduler | cr_stream |
| Use the headless-browser fetch backend | cr_use_browser |
| Use the HTTP fetch backend | cr_use_http |
| Create a crawler | crawler |
| Crawler | Crawler Crawler-class |
| Dataset | Dataset |
| Key-value store | KeyValueStore |
| Request queue | RequestQueue |