{brickster} provides mechanisms to run code against
Databricks, below is an overview of the available of those in the
package:
| Method | Compatible Compute | Notes |
|---|---|---|
| SQL
execution API ( db_sql_exec_*) |
SQL Warehouse | Lower level functions that align 1:1 with API endpoints |
| Command execution context manager ( db_context_manager) |
Clusters (Shared, Single User) |
Higher level R6 class for command execution
contexts. |
| Command
execution API ( db_context_*) |
Clusters (Shared, Single User) |
Lower level functions that align 1:1 with API endpoints |
Databricks
SQL Connector via {reticulate}( db_sql_client) |
SQL Warehouse | Exposes databricks.sql connector to R. |
| Databricks REPL ( db_repl) |
Clusters (Shared, Single User) |
Supports all notebook languages, R is only supported on single user clusters. |
Databricks REPL
(db_repl()) will be the focus of this article.
The REPL temporarily connects the existing R console to a Databricks cluster (via command execution APIs) and allows code in all supported languages to be sent interactively - as if it were running locally.
Using the REPL is simple, to start just provide
cluster_id:
The REPL will check the clusters state and start the cluster if
inactive. The default language is R.
After successfully connecting to the cluster you can run commands against the remote compute from the local session.
The REPL has a shortcut you can enter :<language>
to change the active language. You can change between the following
languages:
| Language | Shortcut |
|---|---|
| R | :r |
| Python | :py |
| SQL | :sql |
| Scala | :scala |
| Shell | :sh |
When you change between languages all variables should persist unless REPL is exited.
There are three addins included with {brickster} for use
with the REPL:
Send selection to console
Send current document to console
Send R script to console
It’s recommended that (1) and (2) be bound to a keyboard shortcut if
using the REPL frequently (‘Tools’ > ‘Modify Keyboard Shortcuts…’
> ‘Search for brickster’).
These addins behave slightly different to the standard shortcuts as they ensure that commands that span multiple lines are handled correctly via the REPL.
Development environments (e.g. RStudio, Positron) won’t display variables from the remote contexts in the environment pane
HTML content will only render for Python,
{htmlwidgets} rendering is restricted due to notebook
limitations that require a workaround currently
Not designed to work with interactive serverless compute
Cannot persist or recover sessions