prefix = FALSE, the default, merge_gtfs() now keeps rows that are identical in every column only once and raises an error when the GTFS objects share ids whose data differ (e.g. a stop_id with different coordinates, or a trip_id with different stop_times), which used to produce inconsistent feeds. Merging a feed with itself now works, but merging versions of the same feed, or feeds with simple ids (e.g. both with an agency_id "1"), usually raises the error: use prefix = TRUE. A character prefix must now have unique elements.filter_by_sf() and the full_trips argument of filter_by_stop_id(), deprecated in version 1.3.0, are now defunct. Use filter_by_spatial_extent(), which takes the same arguments. filter_by_stop_id() now always keeps only the stop_times entries at the given stops (by default, it used to keep entire trips): to keep entire trips, pass the trip_ids that serve the stops to filter_by_trip_id() (#75).get_trip_length() now measures each trip from its first to its last stop, instead of along its entire shape, so lengths are usually shorter, and the speeds of get_trip_speed() and set_trip_speed() slightly lower. Likewise, get_trip_geometry() now returns, by default, the part of the shape between the first and last stops. These functions now require the stops table (and get_trip_length() and get_trip_geometry() also stop_times). Use the new get_shape_length() and get_shapes_geometry() for entire shapes.get_trip_geometry(), get_trip_length() and get_trip_speed() now return one result per trip, without the origin_file column. Their new method argument (and by, in the last two) comes right after trip_id, so unit and file must be passed by name. get_trip_speed() also returns route_id as its first column, and so requires the trips table.by_reference argument of set_trip_speed() must now be passed by name (#89).convert_shapes_to_sf(), get_trip_geometry(), get_trip_length(), get_trip_speed(), get_trip_segment_duration() and get_stop_times_patterns() now always order shapes and stop_times by shape_pt_sequence/stop_sequence, which are therefore required. Results only change for feeds not ordered by these columns, which were previously wrong. Their sort_sequence argument is deprecated: TRUE raises a warning and FALSE an error (#94).convert_shapes_to_sf() and convert_stops_to_sf() are deprecated in favour of get_shapes_geometry() and get_stops_geometry(), which take the same arguments (except sort_sequence) and return the same results.file argument of get_trip_geometry(), get_trip_length() and get_trip_speed() is deprecated in favour of method (file = "stop_times" is method = "euclidean").get_summary() summarises a feed (files, agencies, service calendar, totals including the vehicle-kilometres travelled, and the distributions of trip speeds, trip lengths and route headways), optionally restricted to some dates, routes, modes, agencies and a time of day, as a list of data.tables and an HTML report (requires {ggplot2}, {knitr}, {rmarkdown} and Pandoc). by_route_type = TRUE breaks results down by transport mode (#127), and unit = "mph" reports miles and mph.get_vkt() and get_vmt() return the vehicle-kilometres (or vehicle-miles) travelled by each route, counting every departure (including those in frequencies) on every day its service runs, optionally restricted to some dates and a time of day.get_route_frequency() and get_stop_frequency() return the number of departures and the mean headway of each route and at each stop within a time of day (#53).get_route_speed() returns the minimum, mean, median, maximum and standard deviation of the trip speeds of each route.get_route_geometry() returns the distinct trip geometries of each route (e.g. each direction and short-turn service), with the number of trips that follow each.get_shape_length(), get_shapes_geometry() and get_stops_geometry() return the length of each shape and the shapes and stops as sf objects.get_stops() and get_stop_distances() return the stops visited by the given trips and/or routes and the great-circle distances between each pair of them.get_dates(), get_active_services() and get_start_and_end_times() return the dates with service, the services active on given dates, and the earliest departure and latest arrival times.get_stop_timetable() and get_route_timetable() return the timetables of stops and routes on given dates.get_dwell_time() returns how long vehicles stay at stops, optionally within a time of day.get_calendar_overlap() returns the number of trips each feed runs on each service day and whether all feeds have service on it (or, with resolution = "periods", the periods in which all of them do), to pick a date on which several feeds can be analysed together. plot = TRUE plots the result (requires {ggplot2}). See the new vignette "Checking the calendar overlap of GTFS feeds" (#85). Thanks @higgicd for the suggestion and the original check_gtfs_overlap() code.set_route_frequency() sets the headway of routes within a time of day, replacing their trips in it with a frequency-based template trip. Departures outside the time of day don't change.set_route_speed() sets the average speed of all trips of the given routes, as set_trip_speed() does for trips.set_dwell_time(), set_min_dwell_time() and set_max_speed() set dwell times, raise dwell times at intermediate stops to a minimum and cap trip speeds, shifting the later times of each trip accordingly. The dwell times can be set within a time of day, and speeds capped only between two stops or for trips departing within a time of day, as in set_trip_speed(). Like set_trip_speed() and set_route_speed(), they and set_route_frequency() can be restricted to some transport modes with route_type.interpolate_stop_times() fills blank arrival and departure times, assuming a constant speed between stops with known times.interpolate_shape_dist_traveled() (re)calculates shape_dist_traveled in stop_times and shapes, in meters.stop_times_to_frequencies(), the counterpart of frequencies_to_stop_times(), approximates scheduled trips by frequency-based ones (#69).convert_stops_to_shapes() creates straight-line shapes between consecutive stops for trips without a usable shape.combine_nearby_stops() combines the stops within a given distance of each other (after merging platforms into their stations), updating every file that refers to them. Entrances, generic nodes, boarding areas and pathways are removed.crop_gtfs() crops a feed to a spatial extent. Unlike filter_by_spatial_extent(), which keeps entire trips, it cuts the trips that leave the extent (interpolating blank times at their new ends and shifting their frequencies) and, by default, trims their shapes (#44). Thanks @luukvdmeer for the proposal and discussion.remove_unused_ids() removes unused ids from all files (#55).fix_gtfs() cleans a feed in one call: it removes duplicated rows and unused ids, interpolates blank stop times, recalculates shape_dist_traveled in feeds with shapes, and optionally caps trip speeds and sets a minimum dwell time.build_graph(), which builds a frequency-based transit network from the trips departing within a time of day, as General Modeling Network Specification (GMNS) tables or as an {igraph} or {tidygraph} graph, with in-vehicle, boarding, alighting and walking transfer links. See the new vignette "Building transit network graphs".filter_by_date() and filter_by_period(), which keep the services that run on given dates or in a period. By default (trim = TRUE), filter_by_period() trims the calendars to the period.convert_seconds_to_time(), the counterpart of convert_time_to_seconds().list_validator_versions(). PR contribution by @baarthur.get_trip_length() and get_trip_speed() gain the by argument, to calculate lengths and speeds between consecutive stops. They and get_trip_geometry() gain the method argument, to measure along shapes or along straight lines between stops, and, without shapes, now fall back to straight lines with a warning instead of an error.set_trip_speed() gains first_stop, last_stop, from and to, to change speeds only between two stops and for trips departing within a time of day (#89), and route_type, so trip_id is now optional.get_trip_speed() and set_trip_speed() accept unit = "mph".get_trip_speed(), get_route_speed(), get_trip_duration(), get_trip_length() and get_trip_segment_duration() gain from and to, as in get_dwell_time(), to include only the trips (or segments) that depart within a time of day.frequencies_to_stop_times() gains strategy, to shift departures by half the headway or by a random offset (#56).write_gtfs() gains compression_level. Its default, 6 (previously always 9), makes writing about 2 to 3 times faster, with files of very similar size.path argument of download_validator() now defaults to tempdir().merge_gtfs():
"NA_" when the GTFS objects are passed with names (e.g. merge_gtfs(a = gtfs1, b = gtfs2, prefix = TRUE));field_value and record_sub_id columns of translations, no longer errors with character columns of unknown type, and keeps parent station ids. PR contributions by @gmatosferreira and @haneroglu.merge_gtfs() and remove_duplicates() no longer corrupt elements that are not tables, such as the locations of GTFS-Flex feeds (merge_gtfs() leaves them out, with a warning).download_validator() and list_validator_versions() now fail gracefully on network problems, as CRAN requires: they retry transient failures and then inform with a message (class gtfstools_validator_download_failure) and invisibly return NULL, instead of raising an error (download_validator() used to raise a misleading "invalid URL" one). A version that doesn't exist raises an error of class gtfstools_invalid_validator_version.download_validator() downloads the validator once instead of twice, doesn't go online when an explicit version is already in path, and shows a progress bar with quiet = FALSE. list_validator_versions() returns its result visibly.get_trip_duration(), get_trip_segment_duration(), filter_by_time_of_day(), filter_by_stop_id() and frequencies_to_stop_times() no longer add temporary columns to the tables of the given feed.filter_by_*() functions no longer drop:
fare_rules without route_id, transfers without from_stop_id/to_stop_id and attributions without agency_id;agency of single-agency feeds whose routes.agency_id is blank or absent;fare_attributes entry of feeds without fare_rules. In multi-agency feeds, they keep the fares of the agencies whose routes were kept, and those without an agency.filter_by_agency_id() now keeps the fare_attributes without an agency_id.filter_by_trip_id(), filter_by_service_id(), filter_by_shape_id() and the functions built on them no longer keep fare_attributes (and their agencies) whose fare_rules were all dropped by zone.filter_by_route_id() and filter_by_route_type() filtered fare_rules by level_id instead of route_id.filter_by_time_of_day():
keep = TRUE, sets the end_time of frequencies entries that cross to to one second after it, so frequencies_to_stop_times() no longer loses a departure at to;frequencies entries that start exactly at to (this also affects get_summary());keep = FALSE, drops a departure at to, and removes entries left without departures;start_time of frequencies entries with a blank exact_times, and no longer returns frequency-based trips whose entries were all filtered out;keep = FALSE and full_trips = TRUE, no longer drops every trip with an untimed stop;from/to with minutes or seconds of 60 or more;frequencies table of the given feed when all of its entries are selected.filter_by_spatial_extent() no longer errors when stops have missing coordinates, drops every selected trip with keep = FALSE, no longer keeps unused shapes nor removes duplicated rows, and treats a bbox as a rectangle in longitude and latitude, instead of a slightly larger polygon with curved edges. It no longer accepts sf::st_equals_exact() and sf::st_is_within_distance() as spatial_operation, since they always failed.frequencies_to_stop_times() treats end_time as exclusive, as the GTFS specification requires (from 08:00 to 09:00 every 30 minutes now yields 08:00 and 08:30 only), and sets arrivals that a dwell time at the first stop would put before 00:00:00 to 00:00:00, instead of raising an error."5:30", "12:60:00") and non-ASCII strings marked as "bytes" are converted to NA with a warning, instead of wrong values or an error. convert_time_to_seconds() no longer skips arrival_time and end_time when only one column of a pair is present.get_stop_times_patterns() no longer ignores stop timing in the spatiotemporal patterns of trips with blank stop times, and no longer merges different stop sequences whose stop_ids contain ; or |.set_trip_speed() identifies the first stop of each trip by stop_sequence instead of row position, keeps existing *_secs columns in sync, and raises an error when speed is zero, negative or infinite, or when a trip is given different speeds.get_trip_duration() returns NA with a warning (also raised by get_trip_speed()) for trips without any arrival or departure time, instead of -Inf or an error.shape_id. get_trip_length(), get_trip_speed() and get_trip_geometry() ignore stop_times entries with a missing or blank trip_id, which used to shift the segments of other trips with by = "segment".get_children_stops() no longer returns NA rows for stops without a parent_station.read_gtfs() no longer keeps doubled quotes inside quoted text fields, and write_gtfs() always writes UTF-8, also for feeds read with encoding = "Latin-1".as_dt_gtfs() no longer turns Date fields into NA, and converts the date fields of lists to Date.filter_by_stop_id() now says that it keeps (or drops) the stop_times entries at the given stops, not entire trips.Many functions are faster. The table below shows how many times faster than in version 1.4.0 each function of that version is, on the example feeds poa and spo stacked twice with merge_gtfs() (up to 46,000 stop_times rows), with the default number of threads. Values below 1 mean slower, and ~1 a difference under 1.2 times (measurement noise); functions not listed are ~1 on both feeds, except get_children_stops() and get_parent_station() (the feeds have no parent_station), validate_gtfs() and download_validator(), which were not measured. poa has no frequencies. Functions marked with † now do different work (see "Breaking changes"): merge_gtfs() checks the ids shared by the GTFS objects (with prefix = TRUE, it is 1.2 to 1.8 times faster), and get_trip_geometry() cuts shapes at the trips' first and last stops. On these small feeds, fixed costs dominate: as_dt_gtfs() now converts integer dates (about 1 to 2 milliseconds), filter_by_trip_id() and filter_by_weekday() take about 7 milliseconds more, and the deprecated convert_shapes_to_sf() and convert_stops_to_sf() raise a warning, while their replacements, get_shapes_geometry() and get_stops_geometry(), are about as fast as they were.
| function | n times faster on poa | n times faster on spo |
|---|---|---|
| as_dt_gtfs() | 0.3 | 0.7 |
| convert_sf_to_shapes() | 11 | 65 |
| convert_shapes_to_sf() | 0.1 | 0.4 |
| convert_stops_to_sf() | 0.2 | 0.4 |
| convert_time_to_seconds() | 3.1 | 1.7 |
| filter_by_agency_id() | ~1 | 1.3 |
| filter_by_route_id() | ~1 | 3.3 |
| filter_by_route_type() | ~1 | 2.5 |
| filter_by_shape_id() | 0.7 | 1.3 |
| filter_by_spatial_extent() | 2.5 | 3.1 |
| filter_by_stop_id() | 3.6 | 3.0 |
| filter_by_time_of_day() | 1.4 | 1.7 |
| filter_by_trip_id() | 0.6 | 0.6 |
| filter_by_weekday() | 0.7 | 0.7 |
| frequencies_to_stop_times() | – | 12 |
| get_stop_times_patterns() | 1.4 | ~1 |
| get_trip_duration() | 3.1 | ~1 |
| get_trip_geometry()† | 0.8 | ~1 |
| get_trip_length()† | 1.9 | ~1 |
| get_trip_segment_duration() | 64 | 7.9 |
| get_trip_speed()† | 1.4 | ~1 |
| merge_gtfs()† | 0.1 | 0.1 |
| read_gtfs() | 1.4 | 1.8 |
| set_trip_speed()† | 1.4 | ~1 |
| write_gtfs() | 1.2 | 2.4 |
Most gains grow with feed size: on a feed with 900,000 stop_times rows, convert_time_to_seconds() is about 20 times faster, filter_by_spatial_extent() 25 times, filter_by_time_of_day() 7 times, and get_trip_segment_duration() and get_trip_duration() (when unit isn't "s") 60 and 10 times. merge_gtfs() with prefix is about 2.3 times faster with 1.15 million stop_times rows, get_children_stops() about 250 times faster with 20,000 stops, and read_gtfs() is much faster on feeds with large calendar_dates tables.
get_trip_length(), get_trip_geometry(), get_shape_length(), get_stop_distances() and the functions that use them run their C++ code in parallel, with data.table::getDTthreads() threads (set with data.table::setDTthreads()), or on a single thread where the compiler doesn't support OpenMP. Results don't depend on the number of threads.
Trip lengths are always calculated on the {s2} sphere, regardless of sf::sf_use_s2(). convert_sf_to_shapes() also calculates distances on that sphere: they match the previous ones within a micrometre with sf::sf_use_s2(TRUE) (the default), and differ by up to about 0.4% from the ellipsoidal ones previously calculated with sf::sf_use_s2(FALSE).
gtfstools no longer depends on {units} (#84). set_trip_speed() now converts speeds in m/s or mph with exact factors, so in rare borderline cases a new time may be one second earlier than in version 1.4.0.
download_validator() detects the latest validator version automatically. PR contribution by @baarthur.
Invalid dates (e.g. 20240230) are still converted to NA, but now with a warning.
frequencies_to_stop_times() raises informative errors for invalid frequencies entries and trips without a departure_time.
The warnings about route_ids, trip_ids and stop_ids not found in a feed (classes gtfstools_invalid_route_id, gtfstools_invalid_trip_id and gtfstools_invalid_stop_id) list each missing id once.
The documentation website moved to https://ipea.github.io/gtfstools/ and the GitHub repository to https://github.com/ipea/gtfstools.
download_validator() and validate_gtfs() now support using validator
v5.0.0, v5.0.1 and v6.0.0.{cpp11} functions, as the most recent releases
fix the issues we were having.convert_sf_to_shapes().as_dt_gtfs() with methods for a few different classes
(tidygtfs, gtfs and list).{gtfstools} functions now accepts GTFS objects created by other packages,
such as {gtfsio} and {tidytransit}.filter_by_route_type() now accepts Google Transit's extended route
types.
Thanks @Ge-Rag.convert_shapes_to_sf(), get_trip_geometry(), get_trip_length(),
get_trip_speed(), get_trip_segment_duration() and
get_stop_times_patterns() now take an additional argument sort_sequence,
used to indicate whether shapes/timetables should be ordered by
shape_pt_sequence/stop_sequence when applying the functions' procedures.download_validator() and validate_gtfs() now support using validator
v4.1.0 and v4.2.0.filter_by_stop_id(): include_children and
include_parents.convert_to_standard() in which the date fields from
feed_info would not be converted back to an integer in their standard
format (YYYYMMDD).transfers table based on trip_id
and route_id. Previously they would filter it based only on stop_id.
Thanks Daniel Langbein (@langbein-daniel).filter_by_route_id() in which feeds with only one agency
that omitted agency_id in routes and fare_attributes would end up with
an empty agency table.filter_by_sf() now correctly throws an error when an unsupported function
is passed to spatial_operation.filter_by_stop_id() behavior of filtering by trips that contain the
specified stops has been deprecated. For backwards compatibility reasons,
this behavior is still the default as of the current version and is
controlled by the parameter full_trips. To actually filter by stop ids (the
behavior that we now believe is the most appropriate), please use full_trips = FALSE. This behavior will be the default from version 2.0.0 onward. To
achieve the old behavior, manually subset the stop_times table by stop_id
and specify the trip_ids included in the output in filter_by_trip_id().filter_by_sf() has been deprecated in favor of
filter_by_spatial_extent(). For backwards compatibility reasons, usage of
filter_by_sf() is still allowed as of the current version, but the function
will be removed from the package in version 2.0.0.validate_gtfs() now defaults to run sequentially. Previously it would
default to run parallelly using all available cores. Heavily inspired by
Henrik Bengtsson post "Please Avoid detectCores() in your R packages"
(https://www.jottr.org/2022/12/05/avoid-detectcores/).seconds_to_string() and, consequently, any other
functions that use it.validate_gtfs() behavior. Now used to run MobilityData Canonical GTFS
validator with a feed. The old behavior was marked as deprecated since
v1.0.0.download_validator().write_gtfs() that prevented as_dir = TRUE to be used.set_trip_speed() that resulted in invalid stop_times tables
when max(stop_sequence) was higher than the number of stops of a given
trip. Thanks Alena Stern (@alenastern).set_trip_speed() that resulted in the speed of wrong
trip_ids being updated because of the order that these ids would appear in
the trips and stop_times tables. Thanks Alena Stern (@alenastern).convert_time_to_seconds().filter_by_agency_id().filter_by_service_id().filter_by_time_of_day().filter_by_weekday().frequencies_to_stop_times().get_children_stops().get_stop_times_patterns().get_trip_length().merge_gtfs(): prefix. The warnings parameter was flagged as deprecated.get_parent_station() and get_children_stops now accept stop_id = NULL to analyze all stop_ids in the stops table.get_trip_segment_duration() would list wrong segment numbers, not necessarily starting from 1. Now segment numbers always range from 1 to N, where N is the total number of segments that compose each trip.filter_by_{route,service,shape,trip}_id() that resulted in the agency table not getting filtered when the specified id was character(0).get_trip_geometry(), get_trip_duration(), get_trip_segment_duration() and convert_shapes_to_sf().shape_pt_sequence/stop_sequence in get_trip_geometry() and convert_shapes_to_sf(), since the GTFS reference says that the stop_times and shapes tables must be ordered by point/stop sequence anyway.{lwgeom} from dependencies (Suggests), now that it's not required to run get_trip_speed() and set_trip_speed() anymore.warnings parameter from read_gtfs() and write_gtfs() and the optional and extra parameters from write_gtfs(), flagged as deprecated on gtfstools v1.0.0.convert_stops_to_sf().convert_shapes_to_sf().filter_by_route_type().filter_by_route_id().filter_by_sf().filter_by_shape_id().filter_by_stop_id().filter_by_trip_id().get_parent_station().remove_duplicates().read_gtfs(): fields, skip and encoding. The warnings parameter was flagged as deprecated.write_gtfs(): files, standard_only and as_dir. They substitute the previously existent optional and extra, which were flagged as deprecated. The warnings parameter was flagged as deprecated too.get_trip_speed() and set_trip_speed() examples and tests now only run if {lwgeom} is installed. {lwgeom} is an {sf} "soft" dependency required by these functions, and is listed in Suggests. However, package checks would fail not so gracefully if it wasn't installed, which is now fixed.crs passed to get_trip_geometry() would be assigned to the result without actually reprojecting it.get_trip_geometry() to not raise an error when the 'file' parameter is left untouched and the GTFS object doesn't contain either the shapes or the stop_times table. Closes #29.merge_gtfs() to create objects that inherited only from dt_gtfs (ignoring gtfs and list).get_trip_speed() returned NA speeds if the specified trip_id was listed in trips, but not in stop_times.set_trip_speed() to stop raising a max()-related warning when none of the specified trip_ids exists.{gtfs2gps} will now be exported by {gtfstools}. Huge thanks to the whole {gtfs2gps} crew (Rafael Pereira @rafapereirabr, Pedro Andrade @pedro-andrade-inpe and João Bazzo @Joaobazzo)!{gtfsio}, and many functions now heavily rely on it, such as read_gtfs() and write_gtfs().string_to_seconds() now runs much faster thanks to Mark Padgham (@mpadge).get_trip_geometry() now runs much faster due to data.table-related optimizations.validate_gtfs() will be flagged as deprecated as well, since I plan to heavily change its usability and outputs in future versions.write_gtfs() parameters went through major changes - the optional and extra params were flagged as deprecated and substituted by the more general files and standard_only.