| Title: | Access Novo CAGED Microdata from the Brazilian Ministry of Labour |
|---|---|
| Description: | Download and read the public, non-identified microdata of the Novo CAGED (Cadastro Geral de Empregados e Desempregados), the monthly registry of formal employment movements published by the Brazilian Ministry of Labour and Employment through the PDET FTP server <ftp://ftp.mtps.gov.br/pdet/microdados/>. Lists the reference months available on the server, downloads the three monthly files (movements declared on time, declared late, and exclusions) with an idempotent local cache, and reads the national 7z archives as a stream, filtering by state and selecting columns before anything is kept in memory, so that a single state can be extracted without loading the full national file. Also provides the official record layout and a helper to consolidate admissions, separations and net balance by reference month. |
| Authors: | Andre Leite [aut, cre] (ORCID: <https://orcid.org/0000-0002-4718-9766>), Marcos Wasiliew [aut], Hugo Vasconcelos [aut] (ORCID: <https://orcid.org/0000-0001-6249-0920>), Carlos Amorim [aut] (ORCID: <https://orcid.org/0000-0001-6315-8305>), Diogo Bezerra [aut] (ORCID: <https://orcid.org/0000-0002-1216-8674>), Júlia Nascimento Barreto [aut] |
| Maintainer: | Andre Leite <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.1.0 |
| Built: | 2026-09-30 19:44:09 UTC |
| Source: | https://github.com/cran/cagedr |
Queries the public FTP server of the Ministry of Labour and Employment and
returns the reference months (AAAAMM) of the Novo CAGED that are
currently published, with the date each folder was last modified. The
Ministry publishes a reference month around the end of the following
month, and occasionally re-publishes earlier months.
caged_available(year = NULL, timeout = 30, verbose = NULL)caged_available(year = NULL, timeout = 30, verbose = NULL)
year |
Optional integer vector of years to restrict the listing
(for example |
timeout |
Connection timeout in seconds for each FTP request. |
verbose |
Emit progress messages? Defaults to
|
A tibble with one row per reference month and columns period
(integer AAAAMM), year, month, modified (POSIXct, folder
modification time on the server) and url (the folder URL). Sorted
from the most recent to the oldest month. Returns an empty tibble, with
a warning, when the server cannot be reached.
caged_download() to fetch the files of a month,
caged_periods() to build a sequence of months offline.
# Requires network access to ftp.mtps.gov.br months <- tryCatch(caged_available(year = 2026), error = function(e) NULL) if (!is.null(months)) head(months)# Requires network access to ftp.mtps.gov.br months <- tryCatch(caged_available(year = 2026), error = function(e) NULL) if (!is.null(months)) head(months)
Applies the accounting rule of the Novo CAGED to a set of records read
with caged_read() or caged_fetch() and aggregates them. Movements
declared on time (MOV) and late (FOR) count with their own sign;
exclusions (EXC) cancel a previously declared movement, so their
saldomovimentacao is inverted before summing. The aggregation is always
by competenciamov, the reference month of the movement, which is what
the Ministry's published totals use.
caged_balance(data, by = NULL)caged_balance(data, by = NULL)
data |
A tibble returned by |
by |
Character vector of additional grouping columns (for example
|
Because FOR and EXC archives of month M carry records of earlier
months, the balance of a given month keeps changing as later archives are
published. To reproduce the Ministry's figure for a month, read every
archive published up to the date of interest.
A tibble with the grouping columns and admissions (records
with positive sign after the exclusion rule), separations (negative
sign), net (admissions - separations) and records (number of
records aggregated). When the data contain salario, also
admission_salary (sum of the salaries of the admissions) and
mean_admission_salary.
f <- system.file("extdata", "CAGEDMOV202301_sample.7z", package = "cagedr") x <- caged_read(f, verbose = FALSE) caged_balance(x, by = "uf")f <- system.file("extdata", "CAGEDMOV202301_sample.7z", package = "cagedr") x <- caged_read(f, verbose = FALSE) caged_balance(x, by = "uf")
Remove archives from the cache
caged_cache_clear(period = NULL, cache_dir = NULL)caged_cache_clear(period = NULL, cache_dir = NULL)
period |
Optional reference months ( |
cache_dir |
Optional path. When given, it is returned as is (after creating the folder). |
Invisibly, the number of files removed.
caged_cache_clear()caged_cache_clear()
Downloaded archives are stored in a local cache so that a reference month is never downloaded twice. The location is resolved in this order:
caged_cache_dir(cache_dir = NULL)caged_cache_dir(cache_dir = NULL)
cache_dir |
Optional path. When given, it is returned as is (after creating the folder). |
the cache_dir argument;
the CAGEDR_CACHE_DIR environment variable;
the cagedr.cache_dir R option;
a session-scoped folder under tempdir(), which R removes when the
session ends.
Set one of the first three to keep the archives between sessions. Files published by the Ministry are large (about 55 MB per month) and change only when a month is re-published, so a persistent cache is recommended for repeated work.
The cache directory path, created if needed.
caged_cache_dir() # For a persistent cache, set the environment variable (for instance in # your .Renviron). Here a temporary folder is used and the previous value # is restored afterwards. old <- Sys.getenv("CAGEDR_CACHE_DIR", unset = NA) Sys.setenv(CAGEDR_CACHE_DIR = file.path(tempdir(), "caged-cache")) caged_cache_dir() if (is.na(old)) Sys.unsetenv("CAGEDR_CACHE_DIR") else Sys.setenv(CAGEDR_CACHE_DIR = old)caged_cache_dir() # For a persistent cache, set the environment variable (for instance in # your .Renviron). Here a temporary folder is used and the previous value # is restored afterwards. old <- Sys.getenv("CAGEDR_CACHE_DIR", unset = NA) Sys.setenv(CAGEDR_CACHE_DIR = file.path(tempdir(), "caged-cache")) caged_cache_dir() if (is.na(old)) Sys.unsetenv("CAGEDR_CACHE_DIR") else Sys.setenv(CAGEDR_CACHE_DIR = old)
List the archives currently in the cache
caged_cache_list(cache_dir = NULL)caged_cache_list(cache_dir = NULL)
cache_dir |
Optional path. When given, it is returned as is (after creating the folder). |
A tibble with columns path, period, file (MOV, FOR or
EXC), size_bytes and modified.
caged_cache_list()caged_cache_list()
Fetches the .7z archives of one or more reference months from the
PDET/MTE FTP server into the local cache (see caged_cache_dir()).
Archives already in the cache are not downloaded again unless
force = TRUE.
caged_download( period, files = c("MOV", "FOR", "EXC"), cache_dir = NULL, force = FALSE, timeout = 900, verbose = NULL )caged_download( period, files = c("MOV", "FOR", "EXC"), cache_dir = NULL, force = FALSE, timeout = 900, verbose = NULL )
period |
Reference months in |
files |
Which files to download: any subset of |
cache_dir |
Optional cache directory (see |
force |
Re-download archives already in the cache? |
timeout |
Timeout in seconds for each file. The Ministry's FTP is slow at times; the default allows 15 minutes per file. |
verbose |
Emit progress messages? Defaults to
|
Every reference month has up to three files, all national:
MOV (CAGEDMOV<AAAAMM>.7z): movements declared on time for that
month, the bulk of the data (about 55 MB compressed, 4 to 5 million
records);
FOR (CAGEDFOR<AAAAMM>.7z): movements declared late, which refer
to earlier reference months (competenciamov < AAAAMM);
EXC (CAGEDEXC<AAAAMM>.7z): exclusions, movements cancelled by
the employer, also referring to earlier months.
The first months of the series (January to March 2020) have no FOR or
EXC files; those are reported as not_found, not as errors.
A tibble with one row per (period, file) and columns period,
file, path (local path, NA when not available), status
("downloaded", "cached", "not_found" or "error") and url.
caged_read() to read a downloaded archive, caged_fetch() for
download and read in one call.
# Requires network access; downloads about 1 MB (the EXC file is small). res <- tryCatch( caged_download(202401, files = "EXC", cache_dir = tempdir()), error = function(e) NULL ) res# Requires network access; downloads about 1 MB (the EXC file is small). res <- tryCatch( caged_download(202401, files = "EXC", cache_dir = tempdir()), error = function(e) NULL ) res
Convenience wrapper: caged_download() followed by caged_read() on
every archive found, with the results stacked. Missing files (for
example FOR/EXC in early 2020) are skipped with a message.
caged_fetch( period, uf = NULL, files = c("MOV", "FOR", "EXC"), columns = NULL, cache_dir = NULL, force = FALSE, types = TRUE, chunk_size = 500000L, timeout = 900, verbose = NULL )caged_fetch( period, uf = NULL, files = c("MOV", "FOR", "EXC"), columns = NULL, cache_dir = NULL, force = FALSE, types = TRUE, chunk_size = 500000L, timeout = 900, verbose = NULL )
period |
Reference months in |
uf |
Optional vector of state codes to keep, as IBGE two-digit
numbers (for example |
files |
Which files to download: any subset of |
columns |
Optional character vector of columns to keep, using the
normalized names listed by |
cache_dir |
Optional cache directory (see |
force |
Re-download archives already in the cache? |
types |
Convert the numeric columns of the layout (codes, salary) to
integer/double? If |
chunk_size |
Number of lines parsed per chunk. Larger chunks are faster but use more memory; the default (500,000 lines) uses well under 1 GB. |
timeout |
Timeout in seconds for each file. The Ministry's FTP is slow at times; the default allows 15 minutes per file. |
verbose |
Emit progress messages? Defaults to
|
A tibble with the records of every archive read (see
caged_read() for the columns), or an empty tibble when nothing was
available. The attribute "download" holds the tibble returned by
caged_download(), so that not_found and error files can be
inspected.
# Requires network access. Pernambuco, one month, the three files: pe <- tryCatch( caged_fetch(202401, uf = 26, cache_dir = tempdir(), verbose = FALSE), error = function(e) NULL ) if (!is.null(pe)) table(pe$caged_file)# Requires network access. Pernambuco, one month, the three files: pe <- tryCatch( caged_fetch(202401, uf = 26, cache_dir = tempdir(), verbose = FALSE), error = function(e) NULL ) if (!is.null(pe)) table(pe$caged_file)
The columns of the monthly files, with the original (accented) name used
by the Ministry, the normalized name returned by caged_read(), the type
assigned when types = TRUE, the files in which the column appears and a
short description. The official layout (the "Layout Nao-identificado Novo
Caged Movimentacao" spreadsheet) is published in the same FTP folder as the data.
caged_layout()caged_layout()
Two columns exist only in the exclusions file (EXC): competenciaexc
and indicadordeexclusao.
A tibble with columns column (normalized name), original
(name in the file), type, files and description.
caged_layout() subset(caged_layout(), files != "MOV, FOR, EXC")caged_layout() subset(caged_layout(), files != "MOV, FOR, EXC")
Offline helper that expands a range of reference months in AAAAMM
format. Useful to request several months from caged_download() or
caged_fetch() without querying the server first.
caged_periods(from, to = from)caged_periods(from, to = from)
from, to
|
First and last reference month, as integers or strings in
|
An integer vector of reference months, in increasing order.
caged_periods(202301, 202306) caged_periods("202412")caged_periods(202301, 202306) caged_periods("202412")
Reads the .txt inside a monthly .7z archive without extracting it to
disk and without loading the national file in memory: the text is
decompressed as a stream and parsed in chunks, and each chunk is filtered
by state and reduced to the requested columns before being kept. This is
what makes it practical to extract one state (a few hundred thousand
records) from a national file of several million records on a modest
machine.
caged_read( path, uf = NULL, columns = NULL, types = TRUE, chunk_size = 500000L, verbose = NULL )caged_read( path, uf = NULL, columns = NULL, types = TRUE, chunk_size = 500000L, verbose = NULL )
path |
Path to a |
uf |
Optional vector of state codes to keep, as IBGE two-digit
numbers (for example |
columns |
Optional character vector of columns to keep, using the
normalized names listed by |
types |
Convert the numeric columns of the layout (codes, salary) to
integer/double? If |
chunk_size |
Number of lines parsed per chunk. Larger chunks are faster but use more memory; the default (500,000 lines) uses well under 1 GB. |
verbose |
Emit progress messages? Defaults to
|
A tibble with the selected records and columns, plus two columns
added by the package: caged_file ("MOV", "FOR" or "EXC",
detected from the archive name) and caged_period (the reference month
of the archive, which for FOR and EXC differs from the
competenciamov of the records). Column names are normalized: accents
removed and lower case (competenciamov, municipio, salario).
Returns an empty tibble when no record matches.
caged_layout() for the meaning of every column,
caged_fetch() for download and read in one call.
# A small sample archive ships with the package (Pernambuco and Bahia rows). f <- system.file("extdata", "CAGEDMOV202301_sample.7z", package = "cagedr") x <- caged_read(f, verbose = FALSE) dim(x) # One state, a few columns pe <- caged_read(f, uf = 26, columns = c("competenciamov", "municipio", "saldomovimentacao", "salario"), verbose = FALSE) pe# A small sample archive ships with the package (Pernambuco and Bahia rows). f <- system.file("extdata", "CAGEDMOV202301_sample.7z", package = "cagedr") x <- caged_read(f, verbose = FALSE) dim(x) # One state, a few columns pe <- caged_read(f, uf = 26, columns = c("competenciamov", "municipio", "saldomovimentacao", "salario"), verbose = FALSE) pe