| Title: | Secure and Intuitive Access to 'BigDataPE' 'API' Datasets |
|---|---|
| Description: | Designed to simplify the process of retrieving datasets from the 'Big Data PE' platform using secure token-based authentication. It provides functions for securely storing, retrieving, and managing tokens associated with specific datasets, as well as fetching and processing data. The data-retrieval engine is provided by the generic 'apifetch' package, which 'BigDataPE' configures for the Big Data PE service. |
| Authors: | André Leite [aut, cre] (ORCID: <https://orcid.org/0000-0002-4718-9766>), Hugo Vasconcelos [aut] (ORCID: <https://orcid.org/0000-0001-6249-0920>), Diogo Bezerra [aut] (ORCID: <https://orcid.org/0000-0002-1216-8674>), Marcos Wasiliew [aut] (ORCID: <https://orcid.org/0009-0004-4694-3159>), Carlos Amorim [aut] (ORCID: <https://orcid.org/0000-0001-6315-8305>), Júlia Nascimento Barreto [aut] (ORCID: <https://orcid.org/0009-0004-2851-7770>) |
| Maintainer: | André Leite <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.3.0.9000 |
| Built: | 2026-10-11 11:33:09 UTC |
| Source: | https://github.com/strategicprojects/bigdatape |
This function retrieves data from the BigDataPE API iteratively in chunks.
It calls the API repeatedly with an advancing offset, stopping when a chunk
comes back empty or total_limit is reached, and combines the chunks into a
single tibble (dropping the API's Mensagem status column). If the API
returns more rows than requested, the result is truncated to total_limit.
bdpe_fetch_chunks( base_name, total_limit = Inf, chunk_size = 50000L, query = list(), verbosity = 0L, endpoint = "https://www.bigdata.pe.gov.br/api/buscar" )bdpe_fetch_chunks( base_name, total_limit = Inf, chunk_size = 50000L, query = list(), verbosity = 0L, endpoint = "https://www.bigdata.pe.gov.br/api/buscar" )
base_name |
A string specifying the name of the dataset associated with the token. |
total_limit |
An integer specifying the maximum number of records to fetch. Default is Inf (all available data). |
chunk_size |
An integer specifying the number of records to fetch per chunk. Default is 50000;
|
query |
A named list of additional query parameters to filter the API results. Default is an empty list. |
verbosity |
An integer specifying the verbosity level. Values are:
|
endpoint |
A string specifying the API endpoint URL. Default is "https://www.bigdata.pe.gov.br/api/buscar". |
A tibble containing all the data retrieved from the API.
## Not run: # Store a token for the dataset bdpe_store_token("dengue_dataset", "token") # Fetch up to 500 records in chunks of 100 data <- bdpe_fetch_chunks("dengue_dataset", total_limit = 500, chunk_size = 100) # Fetch all available data in chunks of 200 data <- bdpe_fetch_chunks("dengue_dataset", chunk_size = 200) ## End(Not run)## Not run: # Store a token for the dataset bdpe_store_token("dengue_dataset", "token") # Fetch up to 500 records in chunks of 100 data <- bdpe_fetch_chunks("dengue_dataset", total_limit = 500, chunk_size = 100) # Fetch all available data in chunks of 200 data <- bdpe_fetch_chunks("dengue_dataset", chunk_size = 200) ## End(Not run)
This function retrieves data from the BigDataPE API using securely stored tokens associated with datasets.
Users can specify pagination parameters (limit and offset) and additional query filters to customize the data retrieval.
bdpe_fetch_data( base_name, limit = Inf, offset = 0L, query = list(), verbosity = 0L, endpoint = "https://www.bigdata.pe.gov.br/api/buscar" )bdpe_fetch_data( base_name, limit = Inf, offset = 0L, query = list(), verbosity = 0L, endpoint = "https://www.bigdata.pe.gov.br/api/buscar" )
base_name |
A string specifying the name of the dataset associated with the token. |
limit |
An integer specifying the maximum number of records to retrieve per request. Default is Inf (all records).
If set to a non-positive value or |
offset |
An integer specifying the starting record for the query. Default is 0.
If set to a non-positive value or |
query |
A named list of additional query parameters to filter the API results. Default is an empty list. |
verbosity |
An integer specifying the verbosity level. Values are:
|
endpoint |
A string specifying the API endpoint URL. Default is "https://www.bigdata.pe.gov.br/api/buscar". |
A tibble containing the data returned by the API, without the
API's Mensagem status column.
## Not run: # Store a token for the dataset bdpe_store_token("dengue_dataset", "token") # Fetch 50 records from the beginning data <- bdpe_fetch_data("dengue_dataset", limit = 50) # Fetch records with additional query parameters data <- bdpe_fetch_data("dengue_dataset", query = list(field = "value")) # Fetch all data without limits data <- bdpe_fetch_data("dengue_dataset", limit = Inf) ## End(Not run)## Not run: # Store a token for the dataset bdpe_store_token("dengue_dataset", "token") # Fetch 50 records from the beginning data <- bdpe_fetch_data("dengue_dataset", limit = 50) # Fetch records with additional query parameters data <- bdpe_fetch_data("dengue_dataset", query = list(field = "value")) # Fetch all data without limits data <- bdpe_fetch_data("dengue_dataset", limit = Inf) ## End(Not run)
This function retrieves the authentication token stored
in an environment variable for a specific dataset. If the token is not found,
it returns NULL and prints a warning message instead of throwing an error.
bdpe_get_token(base_name)bdpe_get_token(base_name)
base_name |
The name of the dataset (character). |
A string containing the authentication token, or NULL if the token is not found.
token <- bdpe_get_token("education_dataset")token <- bdpe_get_token("education_dataset")
This function returns a character vector of dataset names
that have their tokens stored in environment variables.
Specifically, it looks for variables that begin with the prefix
"BigDataPE_".
bdpe_list_tokens()bdpe_list_tokens()
A character vector of dataset names with stored tokens. If no tokens are found, an empty vector is returned and a message is printed.
bdpe_list_tokens()bdpe_list_tokens()
This function removes the authentication token stored in an environment variable for a specific dataset. If the token is not found, it prints a message and does not throw an error.
bdpe_remove_token(base_name)bdpe_remove_token(base_name)
base_name |
The name of the dataset (character). |
No return value. If the token is found, it is removed. If not, a message is displayed.
bdpe_remove_token("education_dataset")bdpe_remove_token("education_dataset")
This function stores an authentication token for a specific dataset in a system environment variable. The environment variable name is constructed by converting the dataset name to ASCII (removing accents), replacing spaces with underscores, and prefixing it with "BigDataPE_".
bdpe_store_token(base_name, token, overwrite = FALSE)bdpe_store_token(base_name, token, overwrite = FALSE)
base_name |
The name of the dataset (character). |
token |
The authentication token for the dataset (character). |
overwrite |
Replace an existing token for this dataset? Default |
If a variable with that name already exists (and is non-empty), the function
will not overwrite it unless overwrite = TRUE (e.g. to replace an expired
token).
No return value, called for side effects.
bdpe_store_token("education_dataset", "your-token-here") # Replace an expired token bdpe_store_token("education_dataset", "new-token", overwrite = TRUE) bdpe_remove_token("education_dataset")bdpe_store_token("education_dataset", "your-token-here") # Replace an expired token bdpe_store_token("education_dataset", "new-token", overwrite = TRUE) bdpe_remove_token("education_dataset")
This function appends a list of query parameters to a base URL. It is a thin
re-export of apifetch::parse_queries(). Names and values are URL-encoded;
parameters whose value is NULL, NA or the empty string are dropped; a
parameter with several values is repeated (a=1&a=2); and if url already
has a query string, the new parameters are appended to it.
parse_queries(url, query_list)parse_queries(url, query_list)
url |
The base URL to which query parameters will be added. |
query_list |
A named list of query parameters to be added to the URL. |
The complete URL with the query parameters appended.
parse_queries("https://www.example.com", list(param1 = "value1", param2 = "value2"))parse_queries("https://www.example.com", list(param1 = "value1", param2 = "value2"))