| Title: | Secure and Intuitive Access to 'BigDataPE' 'API' Datasets |
|---|---|
| Description: | Designed to simplify the process of retrieving datasets from the 'BigDataPE' <https://https://is.gd/SQyA8D> platform using secure token-based authentication. It provides functions for securely storing, retrieving, and managing tokens associated with specific datasets, as well as fetching and processing data using the 'httr2' package. The integration with 'keyring' ensures that tokens are stored securely within the system’s credential manager. |
| Authors: | Andre Leite [aut, cre], Hugo Vaconcelos [aut], Diogo Bezerra [aut] |
| Maintainer: | Andre Leite <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.0.9 |
| Built: | 2025-01-17 07:09:21 UTC |
| Source: | https://github.com/strategicprojects/bigdatape |
This function retrieves data from the BigDataPE API iteratively in chunks.
It uses bdpe_fetch_data as the base function and supports limits for the
total number of records to fetch and the size of each chunk.
bdpe_fetch_chunks( base_name, total_limit = Inf, chunk_size = 50000, query = list(), verbosity = 0, endpoint = "https://www.bigdata.pe.gov.br/api/buscar" )bdpe_fetch_chunks( base_name, total_limit = Inf, chunk_size = 50000, query = list(), verbosity = 0, endpoint = "https://www.bigdata.pe.gov.br/api/buscar" )
base_name |
A string specifying the name of the dataset associated with the token. |
total_limit |
An integer specifying the maximum number of records to fetch. Default is Inf (all available data). |
chunk_size |
An integer specifying the number of records to fetch per chunk. Default is 50000 |
query |
A named list of additional query parameters to filter the API results. Default is an empty list. |
verbosity |
An integer specifying the verbosity level for the API requests.
Values are:
- |
endpoint |
A string specifying the API endpoint URL. Default is "https://www.bigdata.pe.gov.br/api/buscar". |
A tibble containing all the data retrieved from the API.
## Not run: # Store a token for the dataset bdpe_store_token("education_dataset", "your-token-here") # Fetch up to 500 records in chunks of 100 data <- bdpe_fetch_chunks("education_dataset", total_limit = 500, chunk_size = 100) # Fetch all available data in chunks of 200 data <- bdpe_fetch_chunks("education_dataset", chunk_size = 200) ## End(Not run)## Not run: # Store a token for the dataset bdpe_store_token("education_dataset", "your-token-here") # Fetch up to 500 records in chunks of 100 data <- bdpe_fetch_chunks("education_dataset", total_limit = 500, chunk_size = 100) # Fetch all available data in chunks of 200 data <- bdpe_fetch_chunks("education_dataset", chunk_size = 200) ## End(Not run)
This function retrieves data from the BigDataPE API using securely stored tokens associated with datasets.
Users can specify pagination parameters (limit and offset) and additional query filters to customize the data retrieval.
bdpe_fetch_data( base_name, limit = Inf, offset = 0, query = list(), verbosity = 0, endpoint = "https://www.bigdata.pe.gov.br/api/buscar" )bdpe_fetch_data( base_name, limit = Inf, offset = 0, query = list(), verbosity = 0, endpoint = "https://www.bigdata.pe.gov.br/api/buscar" )
base_name |
A string specifying the name of the dataset associated with the token. |
limit |
An integer specifying the maximum number of records to retrieve per request. Default is Inf (all records).
If set to a non-positive value or |
offset |
An integer specifying the starting record for the query. Default is 0.
If set to a non-positive value or |
query |
A named list of additional query parameters to filter the API results. Default is an empty list. |
verbosity |
An integer specifying the verbosity level for the API requests.
Values are:
- |
endpoint |
A string specifying the API endpoint URL. Default is "https://www.bigdata.pe.gov.br/api/buscar". |
A tibble containing the data returned by the API.
## Not run: # Store a token for the dataset bdpe_store_token("education_dataset", "your-token-here") # Fetch 50 records from the beginning data <- bdpe_fetch_data("education_dataset", limit = 50) # Fetch records with additional query parameters data <- bdpe_fetch_data("education_dataset", query = list(field = "value")) # Fetch all data without limits data <- bdpe_fetch_data("education_dataset", limit = Inf) ## End(Not run)## Not run: # Store a token for the dataset bdpe_store_token("education_dataset", "your-token-here") # Fetch 50 records from the beginning data <- bdpe_fetch_data("education_dataset", limit = 50) # Fetch records with additional query parameters data <- bdpe_fetch_data("education_dataset", query = list(field = "value")) # Fetch all data without limits data <- bdpe_fetch_data("education_dataset", limit = Inf) ## End(Not run)
This function retrieves the authentication token securely
stored for a specific dataset. If the token is not found,
it returns NULL and prints a message instead of throwing an error.
bdpe_get_token(base_name)bdpe_get_token(base_name)
base_name |
The name of the dataset. |
A string containing the authentication token, or NULL if the token is not found.
token <- bdpe_get_token("education_dataset")token <- bdpe_get_token("education_dataset")
This function returns a list of datasets that have stored tokens in the keyring. If the keyring cannot be accessed (e.g., in a virtual machine or unsupported environment), it returns an empty vector and prints a message.
bdpe_list_tokens()bdpe_list_tokens()
A character vector of dataset names with stored tokens. If the keyring cannot be accessed, an empty vector is returned with a message.
bdpe_list_tokens()bdpe_list_tokens()
This function removes the securely stored authentication token for a specific dataset. If the token is not found, it prints a message and does not throw an error.
bdpe_remove_token(base_name)bdpe_remove_token(base_name)
base_name |
The name of the dataset. |
No return value. If the token is found, it is removed. If not, a message is displayed.
bdpe_remove_token("education_dataset")bdpe_remove_token("education_dataset")
This function securely stores an authentication token
for a specific dataset using the keyring package. If the
keyring is not accessible (e.g., in a virtual machine or unsupported environment),
it prints a message and does not attempt to store the token.
bdpe_store_token(base_name, token)bdpe_store_token(base_name, token)
base_name |
The name of the dataset. |
token |
The authentication token for the dataset. |
No return value. If the keyring is available, the token is securely stored. Otherwise, a message is displayed.
bdpe_store_token("education_dataset", "your-token-here")bdpe_store_token("education_dataset", "your-token-here")
This function appends a list of query parameters to a base URL.
parse_queries(url, query_list)parse_queries(url, query_list)
url |
The base URL to which query parameters will be added. |
query_list |
A list of query parameters to be added to the URL. |
The complete URL with the query parameters appended.
parse_queries("https://www.example.com", list(param1 = "value1", param2 = "value2"))parse_queries("https://www.example.com", list(param1 = "value1", param2 = "value2"))