Added get_task_id_type() and get_task_id_values(), which return the data type of a task ID and the values it takes, from a tasks config, across all rounds or for one round (#266).
get_task_id_names() now returns the task IDs used by every model task of every round. Previously it read only the first model task of each round, so a task ID used only in a later model task was missing from the result and from everything built on it, such as the hub schema and the task ID columns of a <model_out_tbl> (#315).
get_round_idx(), get_round_ids(), get_round_config(), get_round_model_tasks(), get_round_task_id_names(), get_round_output_type_names(), get_task_id_names(), get_derived_task_ids_config(), get_hub_file_formats() and get_hub_derived_task_ids() gain a call argument, so that a function wrapping them can have error messages name the wrapper (#304).
get_round_ids(flatten = "all") now returns unique round IDs, as documented. Previously, a round with round_id_from_variable: true and more than one modeling task contributed one copy of each round ID per modeling task. Errors that list the valid round IDs, such as from get_round_idx(), no longer repeat each ID (#303).
Added read_model_metadata(), which reads a model metadata YAML file. Every array in the file is returned as a list, so a length-1 array is read as a list rather than a string (#299).
Added get_derived_task_ids_config(), which returns the hub or round level derived task IDs from a tasks config object, as get_hub_derived_task_ids() does from a hub path. get_config_derived_task_ids() is an alias (#302).
Added get_round_value_sets_config(), which returns the values each modeling task in a round allows for each task ID and, per output type, for output_type_id (#301).
Added get_round_output_type_names(), which returns the names of the output types a round accepts, as get_round_task_id_names() does for task IDs (#301).
Added get_round_config(), which returns a round's element of the rounds property of the tasks config (#301).
convert_output_type(). Conversions are now computed per group of model/task ID combinations without first materialising an n_samples * n_output_type_ids intermediate table, reducing both runtime and memory allocation by roughly an order of magnitude for large sample tables (#282, thanks @damonbayer for reporting!).target-data.json configuration files (v6.0.0 schema):
get_date_col(): Get the name of the date column across hub data.get_observable_unit(): Get observable unit column names with support for dataset-specific overrides.get_versioned(): Get whether target data is versioned with inheritance from global settings.get_has_output_type_ids(): Get whether oracle-output data has output_type/output_type_id columns.get_non_task_id_schema(): Get the schema for non-task ID columns in time-series data.has_target_data_config() from hubAdmin package to hubUtils. This function checks if a target-data.json file exists in a hub (#260).target-data config files in get_schema_url(), read_config(), get_version_hub(), and is_v3_hub() (#252).testhubs/v6/target_file — target data stored in single files.testhubs/v6/target_dir — hive-partitioned target data.testhubs/v5/target_file — target data stored in single files.testhubs/v5/target_dir — hive-partitioned target data. These hubs are complete and valid, enabling faster examples and checks
without requiring a clone of the public example hub repository.convert_output_type() function to convert model outputs from one output type to another (currently only supports sample to mean, median, and quantile) (#212, #214, #215)convert_output_type() now supports transformations involving output type IDs dependent on task ID variable values (#222)read_config_file now accepts a URL to the raw contents of a JSON config file as well as an object of class <SubTreeFileSystem> pointing to a config file in an S3 cloud hub (#209). This enables reading config files directly from GitHub S3 cloud hubs without having to clone the contents of a hub locally.read_config now also accepts a URL to of a fully configured hub repository hosted on GitHub.is_url(): checks whether a character string is a URL.is_valid_url(): checks whether a URL is valid and reachable.is_github_url(): checks whether a URL is a github.com URL.is_github_repo_url(): checks whether a URL is a GitHub repository URL.create_s3_url(): creates an S3 URL from a bucket name and object path.is_s3_base_fs(): checks whether an object of class <SubTreeFileSystem> is a base file system (i.e. the root of a cloud hub).hubUtils will always only contain released versions of schemas while dev versions from hubUtils (installed from GitHub) may contain versions of schema under active development.subset_task_id_names() function to subset task ID names from a character vector of column names (#149).subset_task_id_cols() and subset_std_cols() to subset a model_out_tbl or submission tbl to task ID or standard (non-task ID) columns respectively (#149).schema_id version checks silenced by default in read_config() and read_config_file().hubValidations functions get_hub_timezone(), get_hub_model_output_dir() and get_hub_file_formats() for extracting hub metadata to hubUtils package.get_hub_derived_task_ids() to extract round or hub level derived task ID values from a tasks.json config file.get_version_config(): extract version from a <config> class object.get_version_config_file(): extract version from a config file by specifying a config_path.get_version_hub(): extract version from a config file by specifying a hub_path.version_equal(): Check whether a schema version property is equal to.version_gte(): Check whether a schema version property is equal to or greater than.version_gt(): Check whether a schema version property is greater than.version_lte(): Check whether a schema version property is equal to or less than.version_lt(): Check whether a schema version property is less than.<config> class objects now have a type attribute to track what type of config they contain (i.e "tasks" or "admin").read_config() and read_config_file() will attempt to coerce their output a <config> class object, with a warning if unsuccessful (#173).as_config() function to coerce a config list to a <config> class object (from the hubAdmin package) (#173).extract_schema_version() where only single digits from each version component were being extracted.get_schema_version_latest() to no longer use v1.0.0hubData dependencyv3 if they are v3.0.0 or above, not just v3.0.0. Thanks to @M-7th for reporting.hubAdmin Suggests dependency by moving test hub configuration validation to CI
(resolved: @annakrystalli, https://github.com/hubverse-org/hubUtils/issues/158)read_config_file() helper function to read a JSON config file from a file path.extract_schema_version() helper function to extract the schema version from a schema id or config schema_version property character string.is_v3_config, is_v3_config_file and is_v3_config_hub to check whether a config object, file or hub is using schema version 3.jsonlite) bug fix.hubUtils package containing significant breaking changes. Much of the package has been moved and split across two smaller and more dedicated packages:
hubData package: contains functions for connecting to and interacting with hub data.
hubData: connect_hub(), connect_model_output(), expand_model_out_val_grid(), create_model_out_submit_tmpl(), coerce_to_character(), coerce_to_hub_schema() and create_hub_schema().hubUtils functions re-exported to hubData: as_model_out_tbl(), validate_model_out_tbl(), model_id_split() and model_id_merge().hubAdmin package: contains functions for administering Hubs, in particular creating and validating hub configuration files. Exported functions moved to hubAdmin:
create_config(), create_model_task(), create_model_tasks(), create_output_type(), create_output_type_cdf(), create_output_type_mean(), create_output_type_median(), create_output_type_pmf(), create_output_type_quantile(), create_output_type_sample(), create_round(), create_rounds(), create_target_metadata(), create_target_metadata_item(), create_task_id(), create_task_ids().validate_config(),validate_model_metadata_schema(), validate_hub_config(), view_config_val_errors().tasks.json config files programmatically (#127).connect_hub() and connect_model_output() now identify and report on files that are present and should have been opened but for which a connection was not successful (#124)validate_model_metadata_schema() function and included it as part of validate_hub_config() (#110 & #112).load_model_metadata() function to compile hub model metadata.coerce_to_character() function for coercing all model output columns to character. This can be much faster than coercing to coerce_to_hub_schema(), especially for dates.expand_model_out_val_grid():
all_character: allow for returning all character columns.as_arrow_table: allow for returning an arrow data table.bind_model_tasks: allow for returning list of model task level grids.expand_model_out_val_grid() when required_vals_only = TRUE yet required task ID columns are not consistent across modeling tasks. The function now pads missing task ID column values with NAs.coerce_to_hub_schema() function and applied it to create_model_out_submit_tmpl() & expand_model_out_val_grid() to ensure column data types in returned tibbles are consistent with the hub's schema (#100).mean/median output types where being included erroneously when required_vals_only = TRUE.get_round_task_id_names() (#99).read_config() (#101).connect_hub() to error when "csv" was an accepted hub file format but there were no CSV in the model output directory. Now connect_hub() checks for the presence of files of each accepted file format and only opens datasets for file formats of which files exists. If there are no files of any accepted file_format in the model output directory, the S3 hub_connection object returned consists of an empty list.hubUtils to be loaded for std_colnames to be internally available.create_model_out_submit_tmpl(). Function now, by default, returns rows of complete cases only and the behavior is controlled by argument complete_cases_only. Argument remove_empty_cols was also removed.create_model_out_submit_tmpl() for generating round specific model output template tibbles (#82).expand_model_out_val_grid() for creating an expanded grid of valid task ID and output type ID across round modeling tasks and output types.get_round_idx(): for getting an integer index of the element in config_tasks$rounds that a character round identifier maps to.get_round_ids(): for getting a list or character vector of Hub round IDs.tasks.json validation checks via validate_config():
required and optional properties.round_id_from_variable is TRUE, check that the specification of the task_id set as round_id is consistent across modeling tasks.round_id values are unique across rounds.std_colnames which contains standard column names used in hubverse model output data files, for use in other hubverse packages (#88).as_model_out_tbl() function to standardize model output data by converting to a model_out_tbl S3 class object. (#32, #33, #63, #64, #66)model_id_merge() and model_id_split() to create model_id column from separate team_abbr and model_abbr columns and vice versa (#63).output_type_id_datatype to connect_hub() to allow overriding default behavior of automatically detecting the output_type_id column data type from the tasks.json config file (#70).create_hub_schema() argument partitions to connect_hub() function to accommodate custom hub partitioning.partition_names to connect_model_output() to accommodate custom hub partitioning.schema to connect_model_output() to allow for overriding default arrow schema auto-detection.jsonvalidate package to Imports so Hub administrator functionality accessible through standard installation.format from create_hub_schema() which now creates the same schema from a tasks.json config file, regardless of the data file format (#80).validate_hub_config() allows maintainers to check the validity of hub config files in a single call. Function view_config_val_errors() also modified to create combined report for hub config files from output of validate_hub_config().model-output data are expected to have output_type & output_type_id instead of type & type_id respectively.connect_hub() now automatically determines the output_type_id column data type from the tasks.json config file coercing to the highest possible data type, "character" being the lowest denominator.create_hub_schema() for determining the schema for data in a hub's model-output directory from a tasks.json config file.connect_hub() now allows establishing connections to hubs with multiple file type formats.create_output_type_categorical() function was renamed to create_output_type_pmf().model-output data directory partitions, was renamed from "model" to "model_id".connect_hub() function to open connection to model-output data
implemented through an arrow FileSystemDataset object. This allows users to create
custom dplyr queries to access model output data.validate_config() function to validate JSON configuration files against Hub
schema as well as function view_config_val_errors() for viewing a concise and easier to
navigate table of validation errors.NEWS.md file to track changes to the package.