Package 'raisr'

List the RAIS archives published on the PDET/MTE FTP server

Description

Queries the public FTP server of the Ministry of Labour and Employment and returns every RAIS archive currently published, one row per file, with its size and the date it was last modified on the server. The Ministry publishes the RAIS of a year around the second half of the following year, sometimes preceded by a partial edition ("parcial"), and keeps the previous files in a Legado folder when a year is re-published ("legado").

Usage

rais_available(year = NULL, timeout = 30, verbose = NULL)

Arguments

year	Optional integer vector of years to restrict the listing (for example 2020:2025). NULL (the default) lists every year since 1985, which takes one FTP request per year folder.
timeout	Connection timeout in seconds for each FTP request.
verbose	Emit progress messages? Defaults to getOption("raisr.verbose", TRUE).

Value

A tibble with one row per archive and columns year, edition ("final", "parcial" or "legado"), type ("vinculos" or "estabelecimentos"), group (the region of a regional file, the state of a pre-2018 file, or NA), file, size_bytes, modified (POSIXct, server time) and url. Sorted from the most recent to the oldest year. Returns an empty tibble, with a warning, when the server cannot be reached.

See Also

rais_files() to build the same table offline for a given year and set of states, rais_download() to fetch the archives.

Examples

# Requires network access to ftp.mtps.gov.br
files <- tryCatch(rais_available(year = 2024), error = function(e) NULL)
if (!is.null(files)) files[, c("edition", "type", "group", "size_bytes")]

---

Remove archives from the cache

Description

Remove archives from the cache

Usage

rais_cache_clear(year = NULL, cache_dir = NULL)

Arguments

year	Optional reference years whose archives are removed. NULL removes every cached archive.
cache_dir	Optional path. When given, it is returned as is (after creating the folder).

Value

Invisibly, the number of files removed.

Examples

rais_cache_clear()

---

Cache directory used by raisr

Description

Downloaded archives are stored in a local cache so that a file is never downloaded twice. Inside the cache, archives keep their server names in one sub-folder per year and edition (2024/RAIS_VINC_PUB_NORDESTE.7z, 2023-parcial/RAIS_ESTAB_PUB.7z, 2017/PE2017.7z). The location is resolved in this order:

Usage

rais_cache_dir(cache_dir = NULL)

Arguments

cache_dir

Optional path. When given, it is returned as is (after creating the folder).

Details

1. the cache_dir argument;

2. the RAISR_CACHE_DIR environment variable;

3. the raisr.cache_dir R option;

4. a session-scoped folder under tempdir(), which R removes when the session ends.

Set one of the first three to keep the archives between sessions. The regional files are large (from about 130 MB for the North region to more than 1 GB for Sao Paulo) and change only when a year is re-published, so a persistent cache is strongly recommended.

Value

The cache directory path, created if needed.

Examples

rais_cache_dir()
## Not run: 
# Persistent cache for every session:
Sys.setenv(RAISR_CACHE_DIR = "~/dados/rais")

## End(Not run)

---

List the archives currently in the cache

Description

List the archives currently in the cache

Usage

rais_cache_list(cache_dir = NULL)

Arguments

cache_dir

Optional path. When given, it is returned as is (after creating the folder).

Value

A tibble with columns path, year, edition, type, group, file, size_bytes and modified.

Examples

rais_cache_list()

---

Download RAIS archives

Description

Fetches from the PDET/MTE FTP server the ⁠.7z⁠ archives that hold the RAIS microdata of one or more years for the requested states, into the local cache (see rais_cache_dir()). The archives to fetch are chosen by the routing rule described in rais_files(): one regional file per group of states from 2018 onwards, one file per state before that, and a national establishments file in both cases. Archives already in the cache are not downloaded again unless force = TRUE.

Usage

rais_download(
  year,
  uf = NULL,
  type = "vinculos",
  edition = "final",
  cache_dir = NULL,
  force = FALSE,
  timeout = 3600,
  verbose = NULL
)

Arguments

year	Reference years (integer or character vector).
uf	Optional states to cover, as IBGE two-digit codes (26) or two-letter abbreviations ("PE"). NULL covers every state (all regional files, or all 27 state files).
type	Which files: "vinculos" (employment relationships, the default), "estabelecimentos" (establishments) or both.
edition	"final" (the default), "parcial" (the preliminary edition some years receive before the final one) or "legado" (the previous files, kept in a Legado sub-folder when a year is re-published).
cache_dir	Optional cache directory (see rais_cache_dir()).
force	Re-download archives already in the cache?
timeout	Timeout in seconds for each file.
verbose	Emit progress messages? Defaults to getOption("raisr.verbose", TRUE).

Details

The files are large: a regional employment file has from about 130 MB (North) to more than 1 GB (Sao Paulo) compressed, and the Ministry's server is slow at times. The default timeout allows one hour per file.

Value

A tibble with one row per archive and columns year, edition, type, group, file, path (local path, NA when not available), status ("downloaded", "cached", "not_found" or "error") and url.

See Also

rais_read() to read a downloaded archive, rais_fetch() for download and read in one call.

Examples

# Which archives would be fetched (offline):
rais_files(2024, uf = "PE", type = c("vinculos", "estabelecimentos"))
## Not run: 
# Pernambuco, RAIS 2024: the NORDESTE regional file (about 600 MB)
rais_download(2024, uf = "PE")

## End(Not run)

---

Download and read RAIS microdata in one call

Description

Convenience wrapper: rais_download() followed by rais_read() on every archive found, with the results stacked. Archives missing on the server are skipped with a message.

Usage

rais_fetch(
  year,
  uf = NULL,
  type = "vinculos",
  edition = "final",
  columns = NULL,
  cache_dir = NULL,
  force = FALSE,
  types = TRUE,
  chunk_size = 500000L,
  timeout = 3600,
  verbose = NULL
)

Arguments

year	Reference years (integer or character vector).
uf	Optional states to cover, as IBGE two-digit codes (26) or two-letter abbreviations ("PE"). NULL covers every state (all regional files, or all 27 state files).
type	Which files: "vinculos" (employment relationships, the default), "estabelecimentos" (establishments) or both.
edition	"final" (the default), "parcial" (the preliminary edition some years receive before the final one) or "legado" (the previous files, kept in a Legado sub-folder when a year is re-published).
columns	Optional character vector of columns to keep, using the normalized names listed by rais_layout(). NULL keeps all columns. municipio is always read (it is needed for filtering) but is only returned when requested or when columns is NULL.
cache_dir	Optional cache directory (see rais_cache_dir()).
force	Re-download archives already in the cache?
types	Convert numeric columns? The Ministry's marker for ignored values (a token in braces, ⁠{n class}⁠ with a tilde on the n) becomes NA in every column; remuneration values and tenure become doubles; codes and counts whose values are all integers become integers; classification codes with leading zeros (CNAE, CBO) stay character. If FALSE every column is returned as character, exactly as in the file (trimmed).
chunk_size	Number of lines parsed per chunk. Larger chunks are faster but use more memory; the default (500,000 lines of 60 columns) uses well under 1 GB.
timeout	Timeout in seconds for each file.
verbose	Emit progress messages? Defaults to getOption("raisr.verbose", TRUE).

Value

A tibble with the records of every archive read (see rais_read() for the columns), or an empty tibble when nothing was available. The attribute "download" holds the tibble returned by rais_download(), so that not_found and error files can be inspected.

Examples

## Not run: 
# Pernambuco, RAIS 2024, a few columns (downloads the 600 MB NORDESTE file)
pe <- rais_fetch(2024, uf = "PE",
                 columns = c("municipio", "cnae_20_subclasse", "vinculo_ativo_31_12",
                             "vl_remun_dezembro_nom"))
rais_stock(pe, by = "municipio")

## End(Not run)

---

Archives needed for a year and a set of states (offline)

Description

Builds, without touching the network, the list of archives that hold the RAIS microdata of one or more years for the requested states. This is the routing rule of the server:

Usage

rais_files(year, uf = NULL, type = "vinculos", edition = "final")

Arguments

year	Reference years (integer or character vector).
uf	Optional states to cover, as IBGE two-digit codes (26) or two-letter abbreviations ("PE"). NULL covers every state (all regional files, or all 27 state files).
type	Which files: "vinculos" (employment relationships, the default), "estabelecimentos" (establishments) or both.
edition	"final" (the default), "parcial" (the preliminary edition some years receive before the final one) or "legado" (the previous files, kept in a Legado sub-folder when a year is re-published).

Details

• from the RAIS 2018 onwards, the employment relationships (vinculos) are published in one file per group of states (RAIS_VINC_PUB_NORTE.7z, ..._NORDESTE.7z, ..._MG_ES_RJ.7z, ..._SP.7z, ..._SUL.7z, ..._CENTRO_OESTE.7z), plus a small RAIS_VINC_PUB_NI.7z with records whose municipality is not identified; the establishments are in a single national RAIS_ESTAB_PUB.7z;

• up to the RAIS 2017, there is one file per state (PE2017.7z, SP2017.7z, ...) and a national ESTB2017.7z.

Value

A tibble with one row per archive and columns year, edition, type, group, file and url.

Examples

rais_files(2024, uf = "PE")
rais_files(2017, uf = c(26, 29), type = c("vinculos", "estabelecimentos"))
rais_files(2024)$file

---

Record layout of the RAIS microdata

Description

The columns of the files, with the normalized name returned by rais_read(), the header used by the Ministry in the ⁠;⁠ files (up to the RAIS 2022, and the partial and legacy editions), the header used in the ⁠,⁠ files (RAIS 2023 onwards), the type assigned when types = TRUE and a short description. The official layouts (one spreadsheet per period, "RAIS_vinculos_layout*.xls" and "RAIS_estabelecimento_layout*.xls") are published in the Layouts folder of the FTP server; the categories of every code are listed there.

Usage

rais_layout(type = "vinculos")

Arguments

type	"vinculos" (employment relationships, the default) or "estabelecimentos" (establishments).

Details

Files before the RAIS 2018 have fewer columns (for example the monthly remuneration columns start in 2015 and are named ⁠vl_rem_<mes>_cc⁠ up to 2017 and ⁠vl_rem_<mes>_sc⁠ from 2018), and the oldest years use a different classification of occupations (cbo_ocupacao) and education (grau_instrucao_2005_1985). Two columns exist only from the RAIS 2023 onwards: ind_vinculo_abandonado and categoria_trabalhador.

Value

A tibble with columns column (normalized name), original (header of the ⁠;⁠ files), original_2023 (header of the ⁠,⁠ files), type and description.

Examples

rais_layout()
rais_layout("estabelecimentos")
subset(rais_layout(), type == "double")$column

---

Read a RAIS archive as a stream

Description

Reads the text file inside a ⁠.7z⁠ archive of the RAIS without extracting it to disk and without loading the whole file in memory: the text is decompressed as a stream and parsed in chunks, and each chunk is filtered by state and reduced to the requested columns before being kept. This is what makes it practical to extract one state (a few million records for a large one) from a regional file of tens of millions of records on a modest machine.

Usage

rais_read(
  path,
  uf = NULL,
  columns = NULL,
  types = TRUE,
  year = NULL,
  chunk_size = 500000L,
  verbose = NULL
)

Arguments

path	Path to an archive, as returned by rais_download().
uf	Optional states to keep, as IBGE two-digit codes (26) or two-letter abbreviations ("PE"). The state is derived from the first two digits of the establishment's municipality code (municipio). NULL keeps every record.
columns	Optional character vector of columns to keep, using the normalized names listed by rais_layout(). NULL keeps all columns. municipio is always read (it is needed for filtering) but is only returned when requested or when columns is NULL.
types	Convert numeric columns? The Ministry's marker for ignored values (a token in braces, ⁠{n class}⁠ with a tilde on the n) becomes NA in every column; remuneration values and tenure become doubles; codes and counts whose values are all integers become integers; classification codes with leading zeros (CNAE, CBO) stay character. If FALSE every column is returned as character, exactly as in the file (trimmed).
year	Reference year of the archive. Detected from the file name (PE2017.7z) or from the folder it sits in (⁠2024/⁠, ⁠2023-legado/⁠), which is how rais_download() lays out the cache; pass it explicitly for a file kept elsewhere.
chunk_size	Number of lines parsed per chunk. Larger chunks are faster but use more memory; the default (500,000 lines of 60 columns) uses well under 1 GB.
verbose	Emit progress messages? Defaults to getOption("raisr.verbose", TRUE).

Details

The function handles the two generations of files published by the Ministry: the ⁠;⁠-separated files with decimal comma (up to the RAIS 2022, and the partial and legacy editions) and the ⁠,⁠-separated files with decimal point published from the RAIS 2023 onwards, whose header names differ. Both are read into the same normalized column names (see rais_layout()), so that years can be stacked.

Value

A tibble with the selected records and columns, plus two columns added by the package: rais_year (the reference year) and rais_type ("vinculos" or "estabelecimentos"). Column names are normalized: accents removed, lower case, words separated by ⁠_⁠, identical across the two header generations. Returns an empty tibble when no record matches.

See Also

rais_layout() for the meaning of every column, rais_fetch() for download and read in one call.

Examples

# Small sample archives ship with the package (Pernambuco and Bahia rows).
f <- system.file("extdata", "2024", "RAIS_VINC_PUB_NORDESTE_sample.7z", package = "raisr")
x <- rais_read(f, verbose = FALSE)
dim(x)

# One state, a few columns
pe <- rais_read(f, uf = "PE",
                columns = c("municipio", "cnae_20_subclasse", "vinculo_ativo_31_12",
                            "vl_remun_media_nom"),
                verbose = FALSE)
pe

---

Consolidate employment stock, admissions and separations

Description

Aggregates employment records read with rais_read() or rais_fetch() into the figures the Ministry publishes from the RAIS: the stock of employment relationships active on 31 December (vinculo_ativo_31_12 == 1), the admissions and separations that happened during the year (records with a non-zero mes_admissao / mes_desligamento) and the December payroll and mean wage of the active stock. The aggregation is by rais_year plus any grouping columns you ask for.

Usage

rais_stock(data, by = NULL)

Arguments

data	A tibble returned by rais_read() or rais_fetch() for vinculos files. Must contain vinculo_ativo_31_12 and rais_year; the other columns are used when present.
by	Character vector of additional grouping columns (for example "municipio", "cnae_20_subclasse", "sexo_trabalhador").

Value

A tibble with the grouping columns and records (records aggregated), stock (relationships active on 31/12), admissions and separations (when mes_admissao / mes_desligamento are present), december_payroll (sum of vl_remun_dezembro_nom over the active stock) and mean_december_wage (that sum divided by the stock with a positive December wage), the last two when vl_remun_dezembro_nom is present.

Examples

f <- system.file("extdata", "2024", "RAIS_VINC_PUB_NORDESTE_sample.7z", package = "raisr")
x <- rais_read(f, verbose = FALSE)
rais_stock(x, by = "municipio")

---

The 27 Brazilian states and the regional file that carries each one

Description

Offline reference table used by the package to translate state codes and abbreviations and to route a state to its regional archive from the RAIS 2018 onwards.

Usage

rais_ufs()

Value

A tibble with columns uf (IBGE two-digit code), sigla (two-letter abbreviation), name and region (the suffix of the ⁠RAIS_VINC_PUB_<region>.7z⁠ file).

Examples

rais_ufs()
subset(rais_ufs(), region == "NORDESTE")

---