Package: crawlee
Title: Tidy Interface for Reproducible Web Crawling
Version: 0.1.1
Authors@R: c(
    person("André", "Leite", email = "leite@castlab.org", role = c("aut", "cre"),
           comment = c(ORCID = "0000-0002-4718-9766")),
    person("Marcos", "Wasiliew", email = "marcos.wasiliew@gmail.com", role = "aut",
           comment = c(ORCID = "0009-0004-4694-3159")),
    person("Hugo", "Vasconcelos", email = "hugo.vasconcelos@ufpe.br", role = "aut",
           comment = c(ORCID = "0000-0001-6249-0920")),
    person("Carlos", "Amorim", email = "carlos.agaf@ufpe.br", role = "aut",
           comment = c(ORCID = "0000-0001-6315-8305")),
    person("Diogo", "Bezerra", email = "diogo.bezerra@ufpe.br", role = "aut",
           comment = c(ORCID = "0000-0002-1216-8674")),
    person("Júlia", "Nascimento Barreto", email = "juliabarreto@gd.seplag.pe.gov.br", role = "aut",
           comment = c(ORCID = "0009-0004-2851-7770")))
Description: A tidy, pipe-friendly toolkit for reproducible web
        crawling and structured data collection, inspired by the
        architecture of the 'Crawlee' library. Provides a unified
        crawler with a deduplicating, resumable request queue,
        content-type aware handlers, structured storage backends and
        rich console logging via 'cli'. Supports crawling HTML pages,
        sitemaps, RSS and Atom feeds and PDF documents, with optional
        headless-browser rendering and helpers for retrieval-augmented
        generation.
License: MIT + file LICENSE
URL: https://github.com/StrategicProjects/crawlee,
        https://strategicprojects.github.io/crawlee/
BugReports: https://github.com/StrategicProjects/crawlee/issues
Depends: R (>= 4.1.0)
Imports: cli, httr2, R6, rlang, rvest, tibble, vctrs, xml2
Suggests: arrow, chromote, DBI, dplyr, duckdb, httptest2, jsonlite,
        knitr, later, nanoparquet, pdftools, promises, rmarkdown,
        testthat (>= 3.0.0)
VignetteBuilder: knitr
Config/testthat/edition: 3
Encoding: UTF-8
Language: en-US
Roxygen: list(markdown = TRUE)
RoxygenNote: 8.0.0
Config/pak/sysreqs: libxml2-dev libssl-dev
Repository: https://strategicprojects.r-universe.dev
Date/Publication: 2026-09-29 14:26:52 UTC
RemoteUrl: https://github.com/strategicprojects/crawlee
RemoteRef: HEAD
RemoteSha: 89889d112baed55955cbd9ebd70ac93dc8c55018
NeedsCompilation: no
Packaged: 2026-09-29 16:11:32 UTC; root
Author: André Leite [aut, cre] (ORCID: <https://orcid.org/0000-0002-4718-9766>),
  Marcos Wasiliew [aut] (ORCID: <https://orcid.org/0009-0004-4694-3159>),
  Hugo Vasconcelos [aut] (ORCID: <https://orcid.org/0000-0001-6249-0920>),
  Carlos Amorim [aut] (ORCID: <https://orcid.org/0000-0001-6315-8305>),
  Diogo Bezerra [aut] (ORCID: <https://orcid.org/0000-0002-1216-8674>),
  Júlia Nascimento Barreto [aut] (ORCID:
    <https://orcid.org/0009-0004-2851-7770>)
Maintainer: André Leite <leite@castlab.org>