Package: crawlee 0.1.0

Andre Leite

crawlee: Tidy Interface for Reproducible Web Crawling

A tidy, pipe-friendly toolkit for reproducible web crawling and structured data collection, inspired by the architecture of the 'Crawlee' library. Provides a unified crawler with a deduplicating, resumable request queue, content-type aware handlers, structured storage backends and rich console logging via 'cli'. Supports crawling HTML pages, sitemaps, RSS and Atom feeds and PDF documents, with optional headless-browser rendering and helpers for retrieval-augmented generation.

Authors:Andre Leite [aut, cre], Marcos Wasilew [aut], Hugo Vasconcelos [aut], Carlos Amorim [aut], Diogo Bezerra [aut]

crawlee_0.1.0.tar.gz
crawlee_0.1.0.zip(r-4.7)crawlee_0.1.0.zip(r-4.6)crawlee_0.1.0.zip(r-4.5)
crawlee_0.1.0.tgz(r-4.6-any)crawlee_0.1.0.tgz(r-4.5-any)
crawlee_0.1.0.tar.gz(r-4.7-any)crawlee_0.1.0.tar.gz(r-4.6-any)
crawlee_0.1.0.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
crawlee/json (API)

# Install 'crawlee' in R:
install.packages('crawlee', repos = c('https://strategicprojects.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/strategicprojects/crawlee/issues

Pkgdown/docs site:https://strategicprojects.github.io

On CRAN:

Conda:

crawleecrawlergovtechhttr2open-datarvestweb-scraping

4.70 score 25 exports 24 dependencies

Last updated from:fef3fd8e7c. Checks:9 OK. Indexed: yes.

TargetResultTimeFilesSyslog
linux-devel-x86_64OK157
source / vignettesOK220
linux-release-x86_64OK159
macos-release-arm64OK84
macos-oldrel-arm64OK81
windows-develOK96
windows-releaseOK86
windows-oldrelOK79
wasm-releaseOK153

Exports:cr_autoscalecr_chunkcr_closecr_collectcr_datasetcr_embedcr_exportcr_from_rsscr_from_sitemapcr_normalize_urlcr_on_htmlcr_on_pdfcr_optionscr_parallelcr_persistcr_runcr_storecr_streamcr_use_browsercr_use_httpcrawlerCrawlerDatasetKeyValueStoreRequestQueue

Dependencies:askpassclicurlgluehttrhttr2jsonlitelifecyclemagrittrmimeopensslpillarpkgconfigR6rappdirsrlangrvestselectrsystibbleutf8vctrswithrxml2

A RAG pipeline
1. Crawl and collect text | 2. Chunk | 3. Embed | 4. Export for retrieval | End to end
Rendered fromrag-pipeline.Rmdusingknitr::rmarkdown

Last update: 2026-06-27
Started: 2026-06-27

Scaling and politeness
Being a good web citizen | Going faster | Fixed-concurrency batches — cr_parallel() | Adaptive batches — cr_autoscale() | Continuous streaming pool — cr_stream() | Choosing an engine | Combining with persistence
Rendered fromscaling-and-politeness.Rmdusingknitr::rmarkdown

Last update: 2026-06-27
Started: 2026-06-27

Storage and resumable runs
The dataset | The key-value store | The request queue and reproducibility | One-call setup: cr_persist() | Resuming
Rendered fromstorage-and-resumable-runs.Rmdusingknitr::rmarkdown

Last update: 2026-06-27
Started: 2026-06-27

Crawling a website
The model | Your first crawler | Following links | Controlling scope | Routing different page types | Crawling from a sitemap | Rendering JavaScript pages | Where next
Rendered fromcrawling-a-website.Rmdusingknitr::rmarkdown

Last update: 2026-06-27
Started: 2026-06-27

Getting started with crawlee
The mental model | A minimal crawl | The handler context | Controlling link discovery | Reproducibility
Rendered fromcrawlee.Rmdusingknitr::rmarkdown

Last update: 2026-06-27
Started: 2026-06-27