| Title: | Utilities for Working with NEON Data |
|---|---|
| Description: | NEON data packages can be accessed through the NEON Data Portal <https://www.neonscience.org> or through the NEON Data API (see <https://data.neonscience.org/data-api> for documentation). Data delivered from the Data Portal are provided as monthly zip files packaged within a parent zip file, while individual files can be accessed from the API. This package provides tools that aid in discovering, downloading, and reformatting data prior to use in analyses. This includes downloading data via the API, merging data tables by type, and converting formats. For more information, see the readme file at <https://github.com/NEONScience/NEON-utilities>. |
| Authors: | Claire Lunch [aut, cre, ctb], Christine Laney [aut, ctb], Nathan Mietkiewicz [aut, ctb], Eric Sokol [aut, ctb], Kaelin Cawley [aut, ctb], NEON (National Ecological Observatory Network) [aut] |
| Maintainer: | Claire Lunch <[email protected]> |
| License: | AGPL-3 |
| Version: | 2.4.3 |
| Built: | 2024-12-05 19:40:22 UTC |
| Source: | CRAN |
Query the API for AOP data by site, year, and product, and download all files found, preserving original folder structure. Downloads serially to avoid overload; may take a very long time.
byFileAOP( dpID, site, year, include.provisional = FALSE, check.size = TRUE, savepath = NA, token = NA_character_ )byFileAOP( dpID, site, year, include.provisional = FALSE, check.size = TRUE, savepath = NA, token = NA_character_ )
dpID |
The identifier of the NEON data product to pull, in the form DPL.PRNUM.REV, e.g. DP1.10023.001 |
site |
The four-letter code of a single NEON site, e.g. 'CLBJ'. |
year |
The four-digit year to search for data. Defaults to 2017. |
include.provisional |
T or F, should provisional data be included in downloaded files? Defaults to F. See https://www.neonscience.org/data-samples/data-management/data-revisions-releases for details on the difference between provisional and released data. |
check.size |
T or F, should the user approve the total file size before downloading? Defaults to T. When working in batch mode, or other non-interactive workflow, use check.size=F. |
savepath |
The file path to download to. Defaults to NA, in which case the working directory is used. |
token |
User specific API token (generated within data.neonscience user accounts) |
A folder in the working directory, containing all files meeting query criteria.
Claire Lunch [email protected] Christine Laney [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
## Not run: # To download 2017 vegetation index data from San Joaquin Experimental Range: byFileAOP(dpID="DP3.30026.001", site="SJER", year="2017") ## End(Not run)## Not run: # To download 2017 vegetation index data from San Joaquin Experimental Range: byFileAOP(dpID="DP3.30026.001", site="SJER", year="2017") ## End(Not run)
Query the API for AOP data by site, year, product, and tile location, and download all files found. Downloads serially to avoid overload; may take a very long time.
byTileAOP( dpID, site, year, easting, northing, buffer = 0, include.provisional = FALSE, check.size = TRUE, savepath = NA, token = NA_character_ )byTileAOP( dpID, site, year, easting, northing, buffer = 0, include.provisional = FALSE, check.size = TRUE, savepath = NA, token = NA_character_ )
dpID |
The identifier of the NEON data product to pull, in the form DPL.PRNUM.REV, e.g. DP1.10023.001 |
site |
The four-letter code of a single NEON site, e.g. 'CLBJ'. |
year |
The four-digit year to search for data. Defaults to 2017. |
easting |
A vector containing the easting UTM coordinates of the locations to download. |
northing |
A vector containing the northing UTM coordinates of the locations to download. |
buffer |
Size, in meters, of the buffer to be included around the coordinates when determining which tiles to download. Defaults to 0. If easting and northing coordinates are the centroids of NEON TOS plots, use buffer=20. |
include.provisional |
T or F, should provisional data be included in downloaded files? Defaults to F. See https://www.neonscience.org/data-samples/data-management/data-revisions-releases for details on the difference between provisional and released data. |
check.size |
T or F, should the user approve the total file size before downloading? Defaults to T. When working in batch mode, or other non-interactive workflow, use check.size=F. |
savepath |
The file path to download to. Defaults to NA, in which case the working directory is used. |
token |
User specific API token (generated within neon.datascience user accounts) |
A folder in the working directory, containing all files meeting query criteria.
Claire Lunch [email protected] Christine Laney [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
A dataset containing NEON data product codes of terrestrial chemistry data products and the "home" data products they are bundled with.
chem_bundleschem_bundles
A data frame with 2 variables:
Data product ID of a terrestrial chemistry product
Data product ID of the corresponding home data product
NEON data product bundles
For any number of bytes, convert to a number of MB or GB
convByteSize(objSize)convByteSize(objSize)
objSize |
The size in bytes |
The size of the file in megabytes or gigabytes
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Create a raster of flux footprint data. Specific to expanded package of eddy covariance data product: DP4.00200.001 For definition of a footprint, see Glossary of Meteorology: https://glossary.ametsoc.org/wiki/Footprint For background information about flux footprints and considerations around the time scale of footprint calculations, see Amiro 1998: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.922.4124&rep=rep1&type=pdf
footRaster(filepath)footRaster(filepath)
filepath |
One of: a folder containing NEON EC H5 files, a zip file of DP4.00200.001 data downloaded from the NEON data portal, a folder of DP4.00200.001 data downloaded by the neonUtilities::zipsByProduct() function, or a single NEON EC H5 file. Filepath can only contain files for a single site. [character] |
Given a filepath containing H5 files of expanded package DP4.00200.001 data, extracts flux footprint data and creates a raster.
A rasterStack object containing all the footprints in the input files, plus one layer (the first in the stack) containing the mean footprint.
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
## Not run: # To run the function on a zip file downloaded from the NEON data portal: ftprnt <- footRaster(filepath="~/NEON_eddy-flux.zip") ## End(Not run)## Not run: # To run the function on a zip file downloaded from the NEON data portal: ftprnt <- footRaster(filepath="~/NEON_eddy-flux.zip") ## End(Not run)
Accesses the API with options to use the user-specific API token generated within neon.datascience user accounts.
getAPI(apiURL, token = NA_character_)getAPI(apiURL, token = NA_character_)
apiURL |
The API endpoint URL |
token |
User specific API token (generated within neon.datascience user accounts). Optional. |
Nate Mietkiewicz [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Accesses the API with options to use the user-specific API token generated within neon.datascience user accounts.
getAPIHeaders(apiURL, token = NA_character_)getAPIHeaders(apiURL, token = NA_character_)
apiURL |
The API endpoint URL |
token |
User specific API token (generated within neon.datascience user accounts). Optional. |
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Most IS products are available at multiple averaging intervals; get a list of what's available for a given data product
getAvg(dpID, token = NA_character_)getAvg(dpID, token = NA_character_)
dpID |
The identifier of the NEON data product, in the form DPL.PRNUM.REV, e.g. DP1.00006.001 |
token |
User specific API token (generated within neon.datascience user accounts) |
A vector of the available averaging intervals, typically in minutes.
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
# Get available averaging intervals for PAR data product getAvg("DP1.00024.001")# Get available averaging intervals for PAR data product getAvg("DP1.00024.001")
Use the DOI Foundation API to get Bibtex-formatted citations for NEON data, or use a template to generate a Bibtex citation for provisional data. Helper function to download and stacking functions.
getCitation(dpID = NA_character_, release = NA_character_)getCitation(dpID = NA_character_, release = NA_character_)
dpID |
The data product ID of the data to be cited [character] |
release |
The data release to be cited. Can be provisional. [character] |
A character string containing the Bibtex citation
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
## Not run: # Get the citation for Breeding landbird point counts (DP1.10003.001), RELEASE-2023 cit <- getCitation(dpID="DP1.10003.001", release="RELEASE-2023") ## End(Not run)## Not run: # Get the citation for Breeding landbird point counts (DP1.10003.001), RELEASE-2023 cit <- getCitation(dpID="DP1.10003.001", release="RELEASE-2023") ## End(Not run)
This is a function to retrieve a data table from the NEON data portal for sites and dates provided by the enduser. NOTE that this function only works for NEON Observation System (OS) data products, and only for select tables
getDatatable( dpid = NA, data_table_name = NA, sample_location_list = NA, sample_location_type = "siteID", sample_date_min = "2012-01-01", sample_date_max = Sys.Date(), sample_date_format = "%Y-%m-%d", data_package_type = "basic", url_prefix_data = "https://data.neonscience.org/api/v0/data/", url_prefix_products = "https://data.neonscience.org/api/v0/products/", token = NA_character_ )getDatatable( dpid = NA, data_table_name = NA, sample_location_list = NA, sample_location_type = "siteID", sample_date_min = "2012-01-01", sample_date_max = Sys.Date(), sample_date_format = "%Y-%m-%d", data_package_type = "basic", url_prefix_data = "https://data.neonscience.org/api/v0/data/", url_prefix_products = "https://data.neonscience.org/api/v0/products/", token = NA_character_ )
dpid |
character sting for NEON data product ID |
data_table_name |
character sting for name of the data table to download, e.g., 'sls_soilCoreCollection' |
sample_location_list |
list of sites, domains, etc. If NA, retrieve all data for the given data table / dpid combination. |
sample_location_type |
character sting for location type, such as 'siteID'. Must be one of the NEON controlled terms. If you're unsure, use 'siteID' |
sample_date_min |
start date for query. Default is 1-Jan-2012, and this should capture the earliest NEON data record. |
sample_date_max |
end date for query. Default is current date. |
sample_date_format |
date format. Default/expected format is yyyy-mm-dd |
data_package_type |
package type, either 'basic' or 'expanded'. If unsure, use 'expanded' |
url_prefix_data |
data endpoint for NEON API. |
url_prefix_products |
products endpoint for NEON API. |
token |
User specific API token (generated within neon.datascience user accounts) |
data frame with selected NEON data
Eric R. Sokol [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Use the NEON API to get the issue log from all products in the bundle in a user-friendly format
getEddyLog(token = NA_character_)getEddyLog(token = NA_character_)
token |
User specific API token (generated within neon.datascience user accounts) |
A table of issues reported for the data product.
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Used to generate a data frame of available AOP files.
getFileUrls(m.urls, include.provisional, token = NA)getFileUrls(m.urls, include.provisional, token = NA)
m.urls |
The monthly API URL for the AOP files |
include.provisional |
T or F, should provisional data be included in downloaded files? |
token |
User specific API token (generated within neon.datascience user accounts) |
A dataframe comprised of file names, S3 URLs, file size, and download status (default = 0)
Claire Lunch [email protected] Christine Laney [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Use the NEON API to get the issue log in a user-friendly format
getIssueLog(dpID = NA, token = NA_character_)getIssueLog(dpID = NA, token = NA_character_)
dpID |
The data product identifier, formatted as DP#.#####.### |
token |
User specific API token (generated within neon.datascience user accounts) |
A table of issues reported for the data product.
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
# Get documentation and availability of plant foliar properties data product cfcIssues <- getIssueLog("DP1.10026.001")# Get documentation and availability of plant foliar properties data product cfcIssues <- getIssueLog("DP1.10026.001")
Use the DataCite API to get NEON data DOIs in a user-friendly format
getNeonDOI(dpID = NA_character_, release = NA_character_)getNeonDOI(dpID = NA_character_, release = NA_character_)
dpID |
The data product identifier, formatted as DP#.#####.### [character] |
release |
Name of a specific release, e.g. RELEASE-2022 [character] |
A table of data product IDs and DOIs.
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
## Not run: # Get all NEON data DOIs allDOIs <- getNeonDOI() ## End(Not run)## Not run: # Get all NEON data DOIs allDOIs <- getNeonDOI() ## End(Not run)
Get a zipped file for a single data product, site, and year-month combination. Use the NEON data portal or API to determine data availability by data product, site, and year-month combinations.
getPackage(dpID, site_code, year_month, package = "basic", savepath = getwd())getPackage(dpID, site_code, year_month, package = "basic", savepath = getwd())
dpID |
The identifier of the NEON data product to pull, in the form DPL.PRNUM.REV, e.g. DP1.10023.001 |
site_code |
A four-letter NEON research site code, such as HEAL for Healy. |
year_month |
The year and month of interest, in format YYYY-MM. |
package |
Either 'basic' or 'expanded', indicating which data package to download. Defaults to basic. |
savepath |
The location to save the output files to |
A zipped monthly file
Christine Laney [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Use the NEON API to get data product information such as availability, science team, etc.
getProductInfo(dpID = "", token = NA)getProductInfo(dpID = "", token = NA)
dpID |
The data product id (optional), formated as DP#.#####.### |
token |
User specific API token (generated within neon.datascience user accounts) |
A named list of metadata and availability information for a single data product. If the dpID argument is omitted, a table of information for all data products in the NEON catalog.
Christine Laney [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
# Get documentation and availability of plant foliar properties data product cfcInfo <- getProductInfo("DP1.10026.001")# Get documentation and availability of plant foliar properties data product cfcInfo <- getProductInfo("DP1.10026.001")
Pull all data from the NEON API /products endpoint, create a data frame with data product ID, data product name, and sensor type.
getProductSensors()getProductSensors()
A data frame
Christine Laney [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
## Not run: sensors <- getProductSensors() ## End(Not run)## Not run: sensors <- getProductSensors() ## End(Not run)
Given a directory, this will recursively list all of the ReadMe files that were unzipped. This should result in a single text file with a list of all of the publication dates from the ReadMe file.
getReadmePublicationDate(savepath, out_filepath, dpID)getReadmePublicationDate(savepath, out_filepath, dpID)
savepath |
The root folder directory where the ReadMe files are located. |
out_filepath |
The output directory and filename. |
dpID |
The data product identifier |
Nathan Mietkiewicz [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Given a list of files, this will order and return the file with the most recent publication date.
getRecentPublication(inList)getRecentPublication(inList)
inList |
The list of files. |
Nathan Mietkiewicz [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
This is a function to retrieve a taxon table from the NEON data portal for the taxon type by the enduser.
getTaxonTable( taxonType = NA, recordReturnLimit = NA, stream = "true", token = NA )getTaxonTable( taxonType = NA, recordReturnLimit = NA, stream = "true", token = NA )
taxonType |
Character string for the taxonTypeCode. Must be one of ALGAE, BEETLE, BIRD, FISH, HERPETOLOGY, MACROINVERTEBRATE, MOSQUITO, MOSQUITO_PATHOGENS, SMALL_MAMMAL, PLANT, TICK |
recordReturnLimit |
Integer. The number of items to limit the result set to. If NA, will return all records in table. |
stream |
Character string, true or false. Option to obtain the result as a stream. Utilize for large requests. |
token |
User specific API token (generated within neon.datascience user accounts) |
data frame with selected NEON data
Eric R. Sokol [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Produces a data frame that is populated by available tiles for the AOP product.
getTileUrls( m.urls, tileEasting, tileNorthing, include.provisional, token = NA_character_ )getTileUrls( m.urls, tileEasting, tileNorthing, include.provisional, token = NA_character_ )
m.urls |
The monthly API URL for the AOP tile. |
tileEasting |
A vector containing the easting UTM coordinates of the locations to download. |
tileNorthing |
A vector containing the northing UTM coordinates of the locations to download. |
include.provisional |
T or F, should provisional data be included in downloaded files? |
token |
User specific API token (generated within neon.datascience user accounts). Optional. |
A dataframe comprised of file names, S3 URLs, file size, and download status (default = 0)
Claire Lunch [email protected] Christine Laney [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Most IS products are available at multiple time intervals; get a list of what's available for a given data product
getTimeIndex(dpID, token = NA_character_)getTimeIndex(dpID, token = NA_character_)
dpID |
The identifier of the NEON data product, in the form DPL.PRNUM.REV, e.g. DP1.00006.001 |
token |
User specific API token (generated within neon.datascience user accounts) |
A vector of the available time intervals, typically in minutes.
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
# Get available time intervals for PAR data product getTimeIndex("DP1.00024.001")# Get available time intervals for PAR data product getTimeIndex("DP1.00024.001")
Extracts a list of table metadata from a single HDF5 file. Specific to eddy covariance data product: DP4.00200.001. Can inform inputs to stackEddy(); variables listed in 'name' are available inputs to the 'var' parameter in stackEddy().
getVarsEddy(filepath)getVarsEddy(filepath)
filepath |
The folder containing the H5 file [character] |
A data frame of the metadata for each data table in the HDF5 file
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
## Not run: # read variables from a file in a hypothetical filepath ec.vars <- getVarsEddy(filepath='/data/NEON.D19.BONA.DP4.00200.001.nsae.2017-12.basic.h5') ## End(Not run)## Not run: # read variables from a file in a hypothetical filepath ec.vars <- getVarsEddy(filepath='/data/NEON.D19.BONA.DP4.00200.001.nsae.2017-12.basic.h5') ## End(Not run)
Used to generate a data frame of available zipfile URLs.
getZipUrls( month.urls, avg, package, dpID, release, messages, tabl, include.provisional, token = NA_character_ )getZipUrls( month.urls, avg, package, dpID, release, messages, tabl, include.provisional, token = NA_character_ )
month.urls |
The monthly API URL for the URL files |
avg |
Global variable for averaging interval |
package |
Global varaible for package type (basic or expanded) |
dpID |
Global variable for data product ID |
release |
Data release to be downloaded |
messages |
Error/warning messages from previous steps |
tabl |
Table name to get |
include.provisional |
Should provisional data be included? |
token |
User specific API token (generated within neon.datascience user accounts) |
A dataframe comprised of file names, S3 URLs, file size, and download status (default = 0)
Claire Lunch [email protected] Christine Laney [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Given the top level zip file, return dataframe of all of the files within it without unzipping the file
listFilesInZip(zippath)listFilesInZip(zippath)
zippath |
The path to a zip file |
A list of filenames within the given zip file
Christine Laney [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Given the data frame of all the files within the top level zip file, return an array of just the zip file names (no pdf, xml, or other files).
listZipfiles(zippath)listZipfiles(zippath)
zippath |
The path to a zip file |
An array of all zip files contained within the focal zip file
Christine Laney [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Pull files from the NEON API, by data product, merge data for each table, and read into the current R environment
loadByProduct( dpID, site = "all", startdate = NA, enddate = NA, package = "basic", release = "current", timeIndex = "all", tabl = "all", check.size = TRUE, include.provisional = FALSE, nCores = 1, forceParallel = FALSE, token = NA_character_, useFasttime = FALSE, avg = NA )loadByProduct( dpID, site = "all", startdate = NA, enddate = NA, package = "basic", release = "current", timeIndex = "all", tabl = "all", check.size = TRUE, include.provisional = FALSE, nCores = 1, forceParallel = FALSE, token = NA_character_, useFasttime = FALSE, avg = NA )
dpID |
The identifier of the NEON data product to pull, in the form DPL.PRNUM.REV, e.g. DP1.10023.001 |
site |
Either the string 'all', meaning all available sites, or a character vector of 4-letter NEON site codes, e.g. c('ONAQ','RMNP'). Defaults to all. |
startdate |
Either NA, meaning all available dates, or a character vector in the form YYYY-MM, e.g. 2017-01. Defaults to NA. |
enddate |
Either NA, meaning all available dates, or a character vector in the form YYYY-MM, e.g. 2017-01. Defaults to NA. |
package |
Either 'basic' or 'expanded', indicating which data package to download. Defaults to basic. |
release |
The data release to be downloaded; either 'current' or the name of a release, e.g. 'RELEASE-2021'. 'current' returns the most recent release, as well as provisional data if include.provisional is set to TRUE. To download only provisional data, use release='PROVISIONAL'. Defaults to 'current'. |
timeIndex |
Either the string 'all', or the time index of data to download, in minutes. Only applicable to sensor (IS) data. Defaults to 'all'. |
tabl |
Either the string 'all', or the name of a single data table to download. Defaults to 'all'. |
check.size |
T or F, should the user approve the total file size before downloading? Defaults to T. When working in batch mode, or other non-interactive workflow, use check.size=F. |
include.provisional |
T or F, should provisional data be included in downloaded files? Defaults to F. See https://www.neonscience.org/data-samples/data-management/data-revisions-releases for details on the difference between provisional and released data. |
nCores |
The number of cores to parallelize the stacking procedure. By default it is set to a single core. |
forceParallel |
If the data volume to be processed does not meet minimum requirements to run in parallel, this overrides. Set to FALSE as default. |
token |
User specific API token (generated within neon.datascience user accounts) |
useFasttime |
Should the fasttime package be used to read date-time fields? Defaults to false. |
avg |
Deprecated; use timeIndex |
All available data meeting the query criteria will be downloaded. Most data products are collected at only a subset of sites, and dates of collection vary. Consult the NEON data portal for sampling details. Dates are specified only to the month because NEON data are provided in monthly packages. Any month included in the search criteria will be included in the download. Start and end date are inclusive.
A named list of all the data tables in the data product downloaded, plus a validation file and a variables file, as available.
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
## Not run: # To download plant foliar properties data from all sites, expanded data package: cfc <- loadByProduct(dpID="DP1.10026.001", site="all", package="expanded") ## End(Not run)## Not run: # To download plant foliar properties data from all sites, expanded data package: cfc <- loadByProduct(dpID="DP1.10026.001", site="all", package="expanded") ## End(Not run)
A dataset containing NEON data product codes of vegetation and sediment data products and the "home" data products they are bundled with.
other_bundlesother_bundles
A data frame with 2 variables:
Data product ID of a product
Data product ID of the corresponding home data product
NEON data product bundles
Used to quiet all output messages
quietMessages(toBeQuieted)quietMessages(toBeQuieted)
toBeQuieted |
Input to be quieted |
The expected output without associated messages/warnings.
Nate Mietkiewicz [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Load a table into R, assigning classes to each column based on data types in variables file; or convert a table already loaded
readTableNEON(dataFile, varFile, useFasttime = FALSE)readTableNEON(dataFile, varFile, useFasttime = FALSE)
dataFile |
A data frame containing a NEON data table, or the filepath to a data table to load |
varFile |
A data frame containing the corresponding NEON variables file, or the filepath to the variables file |
useFasttime |
Should the fasttime package be used to read date-time variables? Defaults to false. |
A data frame of a NEON data table, with column classes assigned by data type
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Given a zipped data file, do a full join of all data files, grouped by table type. This should result in a small number of large files.
stackByTable( filepath, savepath = NA, folder = FALSE, saveUnzippedFiles = FALSE, dpID = NA, package = NA, nCores = 1, useFasttime = FALSE )stackByTable( filepath, savepath = NA, folder = FALSE, saveUnzippedFiles = FALSE, dpID = NA, package = NA, nCores = 1, useFasttime = FALSE )
filepath |
The location of the zip file |
savepath |
The location to save the output files to |
folder |
T or F: does the filepath point to a parent, unzipped folder, or a zip file? If F, assumes the filepath points to a zip file. Defaults to F. No longer needed; included for back compatibility. |
saveUnzippedFiles |
T or F: should the unzipped monthly data folders be retained? |
dpID |
Data product ID of product to stack. Ignored and determined from data unless input is a vector of files, generally via stackFromStore(). |
package |
Data download package, either basic or expanded. Ignored and determined from data unless input is a vector of files, generally via stackFromStore(). |
nCores |
The number of cores to parallelize the stacking procedure. To automatically use the maximum number of cores on your machine we suggest setting nCores=parallel::detectCores(). By default it is set to a single core. |
useFasttime |
Should the fasttime package be used to read date-time variables? Only relevant if savepath="envt". Defaults to false. |
All files are unzipped and one file for each table type is created and written. If savepath="envt" is specified, output is a named list of tables; otherwise, function output is null and files are saved to the location specified.
Christine Laney [email protected] Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
## Not run: # To unzip and merge files downloaded from the NEON Data Portal stackByTable("~/NEON_par.zip") # To unzip and merge files downloaded using zipsByProduct() stackByTable("~/filesToStack00024") ## End(Not run)## Not run: # To unzip and merge files downloaded from the NEON Data Portal stackByTable("~/NEON_par.zip") # To unzip and merge files downloaded using zipsByProduct() stackByTable("~/filesToStack00024") ## End(Not run)
Given a folder of unzipped files (unzipped NEON data file), do a full join of all data files, grouped by table type. This should result in a small number of large files.
stackDataFilesParallel(folder, nCores = 1, dpID)stackDataFilesParallel(folder, nCores = 1, dpID)
folder |
The location of the data |
nCores |
The number of cores to parallelize the stacking procedure. To automatically use the maximum number of cores on your machine we suggest setting 'nCores=parallel::detectCores()'. By default it is set to a single core. If the files are less than 25000 bytes the userdefined nCores will be overridden to a single core. |
dpID |
The data product identifier |
One file for each table type is created and written.
Christine Laney [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Convert data of choice from HDF5 to tabular format. Specific to eddy covariance data product: DP4.00200.001
stackEddy( filepath, level = "dp04", var = NA, avg = NA, metadata = FALSE, useFasttime = FALSE, runLocal = FALSE )stackEddy( filepath, level = "dp04", var = NA, avg = NA, metadata = FALSE, useFasttime = FALSE, runLocal = FALSE )
filepath |
One of: a folder containing NEON EC H5 files, a zip file of DP4.00200.001 data downloaded from the NEON data portal, a folder of DP4.00200.001 data downloaded by the neonUtilities::zipsByProduct() function, or a single NEON EC H5 file [character] |
level |
The level of data to extract; one of dp01, dp02, dp03, dp04 [character] |
var |
The variable set to extract. Can be any of the variables in the "name" level or the "system" level of the H5 file; use the getVarsEddy() function to see the available variables. From the inputs, all variables from "name" and all variables from "system" will be returned, but if variables from both "name" and "system" are specified, the function will return only the intersecting set. This allows the user to, e.g., return only the pressure data ("pres") from the CO2 storage system ("co2Stor"), instead of all the pressure data from all instruments. [character] |
avg |
The averaging interval to extract, in minutes [numeric] |
metadata |
Should the output include metadata from the attributes of the H5 files? Defaults to false. Even when false, variable definitions, issue logs, and science review flags will be included. [logical] |
useFasttime |
Should the fasttime package be used to convert time stamps to time format? Decreases stacking time but can introduce imprecision at the millisecond level. Defaults to false. [logical] |
runLocal |
Set to TRUE to omit any calls to the NEON API. Data are extracted and reformatted from local files, but citation and issue log are not retrieved. [logical] |
Given a filepath containing H5 files of DP4.00200.001 data, extracts variables, stacks data tables over time, and joins variables into a single table. For data product levels 2-4 (dp02, dp03, dp04), joins all available data, except for the flux footprint data in the expanded package. For dp01, an averaging interval and a set of variable names must be provided as inputs.
A named list of data frames. One data frame per site, plus one data frame containing the metadata (objDesc) table and one data frame containing units for each variable (variables).
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
## Not run: # To extract and merge Level 4 data tables, where data files are in the working directory flux <- stackEddy(filepath=getwd(), level='dp04', var=NA, avg=NA) ## End(Not run)## Not run: # To extract and merge Level 4 data tables, where data files are in the working directory flux <- stackEddy(filepath=getwd(), level='dp04', var=NA, avg=NA) ## End(Not run)
Select files from a stored set based on input criteria and pass to stackByTable() or stackEddy()
stackFromStore( filepaths, dpID, site = "all", startdate = NA, enddate = NA, pubdate = NA, timeIndex = "all", level = "dp04", var = NA, zipped = FALSE, package = "basic", load = TRUE, nCores = 1 )stackFromStore( filepaths, dpID, site = "all", startdate = NA, enddate = NA, pubdate = NA, timeIndex = "all", level = "dp04", var = NA, zipped = FALSE, package = "basic", load = TRUE, nCores = 1 )
filepaths |
Either a vector of filepaths pointing to files to be stacked, or a single directory containing files that can be stacked, with selection criteria detmined by the other inputs. In both cases files to be stacked must be either site-month zip files or unzipped folders corresponding to site-month zips. [character] |
dpID |
The NEON data product ID of the data to be stacked [character] |
site |
Either "all" or a vector of NEON site codes to be stacked [character] |
startdate |
Either NA, meaning all available dates, or a character vector in the form YYYY-MM, e.g. 2017-01. Defaults to NA. [character] |
enddate |
Either NA, meaning all available dates, or a character vector in the form YYYY-MM, e.g. 2017-01. Defaults to NA. [character] |
pubdate |
The maximum publication date of data to include in stacking, in the form YYYY-MM-DD. If NA, the most recently published data for each product-site-month combination will be selected. Otherwise, the most recent publication date that is older than pubdate will be selected. Thus the data stacked will be the data that would have been accessed on the NEON Data Portal, if it had been downloaded on pubdate. [character] |
timeIndex |
Either the string 'all', or the time index of data to be stacked, in minutes. Only applicable to sensor (IS) and eddy covariance data. Defaults to 'all'. [character] |
level |
Data product level of data to stack. Only applicable to eddy covariance (SAE) data; see stackEddy() documentation. [character] |
var |
Variables to be extracted and stacked from H5 files. Only applicable to eddy covariance (SAE) data; see stackEddy() documentation. [character] |
zipped |
Should stacking use data from zipped files or unzipped folders? This option allows zips and their equivalent unzipped folders to be stored in the same directory; stacking will extract whichever is specified. Defaults to FALSE, i.e. stacking using unzipped folders. [logical] |
package |
Either "basic" or "expanded", indicating which data package to stack. Defaults to basic. [character] |
load |
If TRUE, stacked data are read into the current R environment. If FALSE, stacked data are written to the directory where data files are stored. Defaults to TRUE. [logical] |
nCores |
Number of cores to use for optional parallel processing. Defaults to 1. [integer] |
If load=TRUE, returns a named list of stacked data tables. If load=FALSE, return is empty and stacked files are written to data directory.
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
A dataset containing publication table names, descriptions, type (site-date, site-all, lab-all, lab-current), and a time index
table_typestable_types
A data frame with 5 variables. Number of rows changes frequently as more tables are added:
data product ID
name of table
description of table
type of table (important for knowing which tables to stack, and how to stack)
a time index (e.g., 0 = native resolution, 1 = 1 minute, 30 = 30 minute averages or totals)
NEON database
Read in a NEON monthly data zip file and parse the respective variables file to create a new GeoCSV file
transformFileToGeoCSV(infile, varfile, outfile)transformFileToGeoCSV(infile, varfile, outfile)
infile |
The path to the file that needs to be parsed |
varfile |
The path to the variables file needed to parse the infile |
outfile |
The path where the new GeoCSV file should be placed |
The same data file with a GeoCSV header
Christine Laney [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Unzip a zip file either at just the top level or recursively through the file
unzipZipfileParallel( zippath, outpath = substr(zippath, 1, nchar(zippath) - 4), level = "all", nCores = 1 )unzipZipfileParallel( zippath, outpath = substr(zippath, 1, nchar(zippath) - 4), level = "all", nCores = 1 )
zippath |
The filepath of the input file |
outpath |
The name of the folder to save unpacked files to |
level |
Whether the unzipping should occur only for the 'top' zip file, or unzip 'all' recursively, or only files 'in' the folder specified |
nCores |
Number of cores to use for parallelization |
Christine Laney [email protected] Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
Pull files from the NEON API, by data product, in a structure that will allow them to be stacked by the stackByTable() function
zipsByProduct( dpID, site = "all", startdate = NA, enddate = NA, package = "basic", release = "current", timeIndex = "all", tabl = "all", check.size = TRUE, include.provisional = FALSE, savepath = NA, load = F, token = NA_character_, avg = NA )zipsByProduct( dpID, site = "all", startdate = NA, enddate = NA, package = "basic", release = "current", timeIndex = "all", tabl = "all", check.size = TRUE, include.provisional = FALSE, savepath = NA, load = F, token = NA_character_, avg = NA )
dpID |
The identifier of the NEON data product to pull, in the form DPL.PRNUM.REV, e.g. DP1.10023.001 |
site |
Either the string 'all', meaning all available sites, or a character vector of 4-letter NEON site codes, e.g. c('ONAQ','RMNP'). Defaults to all. |
startdate |
Either NA, meaning all available dates, or a character vector in the form YYYY-MM, e.g. 2017-01. Defaults to NA. |
enddate |
Either NA, meaning all available dates, or a character vector in the form YYYY-MM, e.g. 2017-01. Defaults to NA. |
package |
Either 'basic' or 'expanded', indicating which data package to download. Defaults to basic. |
release |
The data release to be downloaded; either 'current' or the name of a release, e.g. 'RELEASE-2021'. 'current' returns the most recent release, as well as provisional data if include.provisional is set to TRUE. To download only provisional data, use release='PROVISIONAL'. Defaults to 'current'. |
timeIndex |
Either the string 'all', or the time index of data to download, in minutes. Only applicable to sensor (IS) data. Defaults to 'all'. |
tabl |
Either the string 'all', or the name of a single data table to download. Defaults to 'all'. |
check.size |
T or F, should the user approve the total file size before downloading? Defaults to T. When working in batch mode, or other non-interactive workflow, use check.size=F. |
include.provisional |
T or F, should provisional data be included in downloaded files? Defaults to F. See https://www.neonscience.org/data-samples/data-management/data-revisions-releases for details on the difference between provisional and released data. |
savepath |
The location to save the output files to |
load |
T or F, are files saved locally or loaded directly? Used silently with loadByProduct(), do not set manually. |
token |
User specific API token (generated within neon.datascience user accounts). Optional. |
avg |
Deprecated; use timeIndex |
All available data meeting the query criteria will be downloaded. Most data products are collected at only a subset of sites, and dates of collection vary. Consult the NEON data portal for sampling details. Dates are specified only to the month because NEON data are provided in monthly packages. Any month included in the search criteria will be included in the download. Start and end date are inclusive. timeIndex: NEON sensor data are published at pre-determined averaging intervals, usually 1 and 30 minutes. The default download includes all available data. Download volume can be greatly reduced by downloading only the 30 minute files, if higher frequency data are not needed. Use getTimeIndex() to find the available averaging intervals for each sensor data product.
A folder in the working directory (or in savepath, if specified), containing all zip files meeting query criteria.
Claire Lunch [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
## Not run: # To download plant foliar properties data from all sites, expanded data package: zipsByProduct(dpID="DP1.10026.001", site="all", package="expanded") ## End(Not run)## Not run: # To download plant foliar properties data from all sites, expanded data package: zipsByProduct(dpID="DP1.10026.001", site="all", package="expanded") ## End(Not run)
Read in a set of URLs from NEON data tables and then download the data from the NEON ECS buckets. Assumes data tables are in the format resulting from merging files using stackByTable(). File downloads from ECS can be extremely large; be prepared for long download times and large file storage.
zipsByURI( filepath, savepath = paste0(filepath, "/ECS_zipFiles"), pick.files = FALSE, check.size = TRUE, unzip = TRUE, saveZippedFiles = FALSE, token = NA_character_ )zipsByURI( filepath, savepath = paste0(filepath, "/ECS_zipFiles"), pick.files = FALSE, check.size = TRUE, unzip = TRUE, saveZippedFiles = FALSE, token = NA_character_ )
filepath |
The location of the NEON data containing URIs. Can be either a local directory containing NEON tabular data or a list object containing tabular data. |
savepath |
The location to save the output files from the ECS bucket, optional. Defaults to creating a "ECS_zipFiles" folder in the filepath directory. |
pick.files |
T or F, should the user be told the name of each file before downloading? Defaults to F. When working in batch mode, or other non-interactive workflow, use pick.files=F. |
check.size |
T or F, should the user be told the total file size before downloading? Defaults to T. When working in batch mode, or other non-interactive workflow, use check.size=F. |
unzip |
T or F, indicates if the downloaded zip files from ECS buckets should be unzipped into the same directory, defaults to T. Supports .zip and .tar.gz files currently. |
saveZippedFiles |
T or F: should the zip files be retained after unzipping? Defaults to F. |
token |
User specific API token (generated within neon.datascience user accounts). Optional. |
A folder in the working directory (or in savepath, if specified), containing all files meeting query criteria.
Kaelin Cawley [email protected]
License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007
## Not run: # To download stream morphology data from stacked data: zipsByURI(filepath="~/filesToStack00131/stackedFiles") ## End(Not run)## Not run: # To download stream morphology data from stacked data: zipsByURI(filepath="~/filesToStack00131/stackedFiles") ## End(Not run)