6. Choosing a HTTP request class

There are a number of different crul classes that do HTTP requests. The following table compares features across the classes.

HttpClient is the main class for doing synchronous HTTP requests. It supports all HTTP verbs including retry. It was the first class in this package. See also crul::ok(), which builds on this class.

Paginator is a class for automating pagination. It requires an instance of HttpClient as it’s first parameter. It does not handle asynchronous requests at this time, but may in the future. Paginator may be the right class to use when you don’t know the total number of results. Beware however, that if there are A LOT of results (and a lot depends on your internet speed and the server response time) the requests may take a long time to finish - just plan wisely to fit your needs.

ok is a convienence function light wrapper around HttpClient. It’s use case is for determining if a URL is “up”, or “okay”. You don’t have to instantiate an R6 class as you do with the other classes discussed here, but you can pass an HttpClient class to it if you like.

With Async you can make HTTP requests in parallel. Async does not at this time support retry. It’s targeted at the use case where you don’t mind having request settings the same for all requests - you just pass in any number of URLs and all requests get the same headers, auth, curl options applied, if any.

AsyncVaried allows you to customize each request using HttpRequest (See below); that is, every HTTP request run asynchronously can have its own curl options, headers, etc.

AsyncQueue is the newest class, inheriting from AsyncVaried, introduced in curl v1. The motivation behind this class is: sometimes you may want to do HTTP requests in parallel, but there’s rate limiting or some other reason to want to not simply send off all requests immediately. AsyncQueue sets up a queue, splitting up requests into buckets, and executes requests based on a sleep or req_per_min (requests per minute) setting.