Package: crawlee 0.1.1

André Leite

crawlee: Tidy Interface for Reproducible Web Crawling

A tidy, pipe-friendly toolkit for reproducible web crawling and structured data collection, inspired by the architecture of the 'Crawlee' library. Provides a unified crawler with a deduplicating, resumable request queue, content-type aware handlers, structured storage backends and rich console logging via 'cli'. Supports crawling HTML pages, sitemaps, RSS and Atom feeds and PDF documents, with optional headless-browser rendering and helpers for retrieval-augmented generation.

Authors:André Leite [aut, cre], Marcos Wasiliew [aut], Hugo Vasconcelos [aut], Carlos Amorim [aut], Diogo Bezerra [aut], Júlia Nascimento Barreto [aut]

crawlee_0.1.1.tar.gz
crawlee_0.1.1.zip(r-4.7-any)crawlee_0.1.1.zip(r-4.6-any)crawlee_0.1.1.zip(r-4.5-any)
crawlee_0.1.1.tgz(r-4.6-any)crawlee_0.1.1.tgz(r-4.5-any)
crawlee_0.1.1.tar.gz(r-4.7-any)crawlee_0.1.1.tar.gz(r-4.6-any)
crawlee_0.1.1.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
crawlee/json (API)

# Install 'crawlee' in R:
install.packages('crawlee', repos = c('https://strategicprojects.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/strategicprojects/crawlee/issues

Pkgdown/docs site:https://strategicprojects.github.io

On CRAN:

Conda:

crawleecrawlergovtechhttr2open-datarvestweb-scraping

5.18 score 2 stars 379 downloads 25 exports 23 dependencies

Last updated from:89889d112b. Checks:9 OK. Indexed: yes.

TargetResultTimeFilesSyslog
linux-develOK152
source / vignettesOK218
linux-releaseOK139
macos-releaseOK107
macos-oldrelOK90
windows-develOK2931
windows-releaseOK2450
windows-oldrelOK2957
wasm-releaseOK137

Exports:cr_autoscalecr_chunkcr_closecr_collectcr_datasetcr_embedcr_exportcr_from_rsscr_from_sitemapcr_normalize_urlcr_on_htmlcr_on_pdfcr_optionscr_parallelcr_persistcr_runcr_storecr_streamcr_use_browsercr_use_httpcrawlerCrawlerDatasetKeyValueStoreRequestQueue

Dependencies:askpassclicurlgluehttrhttr2jsonlitelifecyclemagrittrmimeopensslpillarpkgconfigR6rlangrvestselectrsystibbleutf8vctrswithrxml2

A RAG pipeline
1. Crawl and collect text | 2. Chunk | 3. Embed | 4. Export for retrieval | End to end
Rendered fromrag-pipeline.Rmdusingknitr::rmarkdown

Last update: 2026-06-27
Started: 2026-06-27

Scaling and politeness
Being a good web citizen | Going faster | Fixed-concurrency batches — cr_parallel() | Adaptive batches — cr_autoscale() | Continuous streaming pool — cr_stream() | Choosing an engine | Combining with persistence
Rendered fromscaling-and-politeness.Rmdusingknitr::rmarkdown

Last update: 2026-06-27
Started: 2026-06-27

Storage and resumable runs
The dataset | The key-value store | The request queue and reproducibility | One-call setup: cr_persist() | Resuming
Rendered fromstorage-and-resumable-runs.Rmdusingknitr::rmarkdown

Last update: 2026-06-27
Started: 2026-06-27

Crawling a website
The model | Your first crawler | Following links | Controlling scope | Routing different page types | Crawling from a sitemap | Rendering JavaScript pages | Where next
Rendered fromcrawling-a-website.Rmdusingknitr::rmarkdown

Last update: 2026-06-27
Started: 2026-06-27

Getting started with crawlee
The mental model | A minimal crawl | The handler context | Controlling link discovery | Reproducibility
Rendered fromcrawlee.Rmdusingknitr::rmarkdown

Last update: 2026-06-27
Started: 2026-06-27