| Title: | Dummy R APIs Used in 'Azure Synapse Analytics' for Local Developments |
|---|---|
| Description: | This is a pure dummy interfaces package which mirrors 'MsSparkUtils' APIs <https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/microsoft-spark-utilities?pivots=programming-language-r> of 'Azure Synapse Analytics' <https://learn.microsoft.com/en-us/azure/synapse-analytics/> for R users, customer of Azure Synapse can download this package from CRAN for local development. |
| Authors: | runtimeexp [aut, cre], Microsoft [cph] |
| Maintainer: | runtimeexp <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 1.6.2 |
| Built: | 2026-07-03 21:29:40 UTC |
| Source: | https://github.com/cran/notebookutils |
Set the dataframe info which needs to be visualized.
display(dataFrame, isSummary = FALSE)display(dataFrame, isSummary = FALSE)
dataFrame |
the dataframe that needs to be visualized. |
isSummary |
whether show summary info of the dataframe. |
No return value, show the first part of passed dataFrame.
data <- list(56,78,90,45,67) df <- data.frame(t(sapply(data,c))) display(df) display(df, TRUE)data <- list(56,78,90,45,67) df <- data.frame(t(sapply(data,c))) display(df) display(df, TRUE)
display) which needs to be visualized.Set the chart config metadata for current dataframe (set by display) which needs to be visualized.
display.config( commId, lastCommId = NULL, binsNumber = 10, category = "table", keys = NULL, values = NULL, series = NULL, aggregation = NULL, column = NULL )display.config( commId, lastCommId = NULL, binsNumber = 10, category = "table", keys = NULL, values = NULL, series = NULL, aggregation = NULL, column = NULL )
commId |
the id used to identify whether the API call from synapse notebook js client. |
lastCommId |
same with |
binsNumber |
bins number for rendering histogram, default is 10. |
category |
the chart category as |
keys |
the column names which useds to render x-axis. |
values |
the column names which used to render y-axis. |
series |
the column which used to render the chart series |
aggregation |
the aggregation operation type: |
column |
will deperated: the column name used to calculate the statistic info, as the column type, unique values, missing values, etc. |
display.Set the visualizer custom options related to the current dataframe, it will be consumed when executing display.
display.configCustomOptions(options)display.configCustomOptions(options)
options |
custom options serialized from JSON format |
Construct an specific html fragment to synapse notebook front-end for rendering based on user-input html content.
displayHTML(content)displayHTML(content)
content |
html content which user want to render |
No return value, print the content to mimic the render behavior when used in azure synapse runtime.
displayHTML('<b>Hello world!</b>')displayHTML('<b>Hello world!</b>')
Take linked service name as input and return connection string or credentials depending on the configuration of the linked service.
mssparkutils.credentials.getConnectionStringOrCreds(linkedService)mssparkutils.credentials.getConnectionStringOrCreds(linkedService)
linkedService |
Linked service name. |
A empty string used to mimic credentials returned by azure synapse runtime for linkedService.
mssparkutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')mssparkutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')
Take linked service name as input and return full connection string with credentials.
mssparkutils.credentials.getFullConnectionString(linkedService)mssparkutils.credentials.getFullConnectionString(linkedService)
linkedService |
Linked service name. |
A empty string used to mimic connection string returned by azure synapse runtime for linkedService.
mssparkutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')mssparkutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')
Return all the properties of a given linked service in string format.
mssparkutils.credentials.getPropertiesAll(linkedService)mssparkutils.credentials.getPropertiesAll(linkedService)
linkedService |
Linked service name. |
A empty string used to mimic properties string returned by azure synapse runtime for linkedService.
mssparkutils.credentials.getPropertiesAll('AzureDataLakeStorage1')mssparkutils.credentials.getPropertiesAll('AzureDataLakeStorage1')
Return AKV secret.
mssparkutils.credentials.getSecret(akvName, secret, linkedService = NULL)mssparkutils.credentials.getSecret(akvName, secret, linkedService = NULL)
akvName |
Azure Key Vault name. |
secret |
name of the secret being fetched. |
linkedService |
linkedService name of the AKV linked service. |
A empty string used to mimic secret returned by azure synapse runtime for given akvName and secret.
mssparkutils.credentials.getSecret('akvName', 'secretName') mssparkutils.credentials.getSecret('akvName', 'secretName', 'AzureDataLakeStorage1')mssparkutils.credentials.getSecret('akvName', 'secretName') mssparkutils.credentials.getSecret('akvName', 'secretName', 'AzureDataLakeStorage1')
Return AKV secret using linkedService.
mssparkutils.credentials.getSecretWithLS(linkedService, secret)mssparkutils.credentials.getSecretWithLS(linkedService, secret)
linkedService |
linkedService name of the AKV linked service. |
secret |
name of the secret being fetched. |
A empty string used to mimic secret returned by azure synapse runtime for given linkedService and secret.
mssparkutils.credentials.getSecretWithLS('AzureDataLakeStorage1', 'secretName')mssparkutils.credentials.getSecretWithLS('AzureDataLakeStorage1', 'secretName')
Get AAD token for a resource.
mssparkutils.credentials.getToken(audience, name = "")mssparkutils.credentials.getToken(audience, name = "")
audience |
token audience. |
name |
token audience. |
A empty string used to mimic token returned by azure synapse runtime for accessing resource audience.
mssparkutils.credentials.getToken('synapse') mssparkutils.credentials.getToken('storage') mssparkutils.credentials.getToken('storage', 'storage')mssparkutils.credentials.getToken('synapse') mssparkutils.credentials.getToken('storage') mssparkutils.credentials.getToken('storage', 'storage')
Get help message.
mssparkutils.credentials.help()mssparkutils.credentials.help()
No return value, print empty string to mimic the behavior of help method of mssparkutils credentials module when used in azure synapse runtime.
mssparkutils.credentials.help()mssparkutils.credentials.help()
Returns true if the input token is valid (i.e, hasn't expired).
mssparkutils.credentials.isValidToken(token)mssparkutils.credentials.isValidToken(token)
token |
token to validate. |
FALSE to mimic the result if token is invalid.
mssparkutils.credentials.isValidToken('dummyToken')mssparkutils.credentials.isValidToken('dummyToken')
Put AKV secret using with or without linkedService.
mssparkutils.credentials.putSecret( akvName, secretName, secretValue, linkedService = NULL )mssparkutils.credentials.putSecret( akvName, secretName, secretValue, linkedService = NULL )
akvName |
Azure Key Vault name. |
secretName |
name of the secret being written. |
secretValue |
value of the secret being written. |
linkedService |
name of the AKV linked service. |
The secretValue been written.
mssparkutils.credentials.putSecret('akvName', 'secretName', 'secretValue') mssparkutils.credentials.putSecret('akvName', 'secretName', 'secretValue', 'AzureDataLakeStorage1')mssparkutils.credentials.putSecret('akvName', 'secretName', 'secretValue') mssparkutils.credentials.putSecret('akvName', 'secretName', 'secretValue', 'AzureDataLakeStorage1')
Put AKV secret using linkedService.
mssparkutils.credentials.putSecretWithLS( linkedService, secretName, secretValue )mssparkutils.credentials.putSecretWithLS( linkedService, secretName, secretValue )
linkedService |
name of AKV linked service. |
secretName |
name of the secret being written. |
secretValue |
value of the secret being written. |
The secretValue been written.
mssparkutils.credentials.putSecretWithLS('AzureDataLakeStorage1', 'secretName', 'secretValue')mssparkutils.credentials.putSecretWithLS('AzureDataLakeStorage1', 'secretName', 'secretValue')
Get cluster id.
mssparkutils.env.getClusterId()mssparkutils.env.getClusterId()
A empty string used to mimic cluster id of azure synapse runtime.
mssparkutils.env.getClusterId()mssparkutils.env.getClusterId()
Get job Id.
mssparkutils.env.getJobId()mssparkutils.env.getJobId()
A empty string used to mimic the id of spark job been submitted to azure synapse runtime.
mssparkutils.env.getJobId()mssparkutils.env.getJobId()
Get pool name.
mssparkutils.env.getPoolName()mssparkutils.env.getPoolName()
A empty string used to mimic the name of user's azure synapse spark pool.
mssparkutils.env.getPoolName()mssparkutils.env.getPoolName()
Get user Id.
mssparkutils.env.getUserId()mssparkutils.env.getUserId()
A empty string used to mimic the id of user.
mssparkutils.env.getUserId()mssparkutils.env.getUserId()
Get user name.
mssparkutils.env.getUserName()mssparkutils.env.getUserName()
A empty string used to mimic the name of user.
mssparkutils.env.getUserName()mssparkutils.env.getUserName()
Get workspace name.
mssparkutils.env.getWorkspaceName()mssparkutils.env.getWorkspaceName()
A empty string used to mimic the id of the user's azure synapse workspace.
mssparkutils.env.getWorkspaceName()mssparkutils.env.getWorkspaceName()
Get help message.
mssparkutils.env.help()mssparkutils.env.help()
No return value, print empty string to mimic the behavior of help method of mssparkutils env module when used in azure synapse runtime.
mssparkutils.env.help()mssparkutils.env.help()
Append the given String to a file, encoded in UTF-8.
mssparkutils.fs.append(file, content, createFileIfNotExists = FALSE)mssparkutils.fs.append(file, content, createFileIfNotExists = FALSE)
file |
FileSystem URI |
content |
Content needs to be append to file, encoded in System default charset. |
createFileIfNotExists |
If set to true, will firstly try to create file if not exists. |
FALSE to mimic the result if file content append fail.
mssparkutils.fs.append("/tmp/my-file", "Hello world!") mssparkutils.fs.append("/tmp/my-file", "Hello world!", TRUE)mssparkutils.fs.append("/tmp/my-file", "Hello world!") mssparkutils.fs.append("/tmp/my-file", "Hello world!", TRUE)
Copies a file or directory, possibly across FileSystems.
mssparkutils.fs.cp(from, to, recurse = FALSE)mssparkutils.fs.cp(from, to, recurse = FALSE)
from |
FileSystem URI of the source file or directory |
to |
FileSystem URI of the destination file or directory |
recurse |
if TRUE, all files and directories will be recursively copied |
FALSE to mimic the result if file or directory from fail to copy to to.
mssparkutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b") mssparkutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b", TRUE)mssparkutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b") mssparkutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b", TRUE)
Check if a file or directory exists.
mssparkutils.fs.exists(file)mssparkutils.fs.exists(file)
file |
FileSystem URI |
TRUE if the file or directory exists
## Not run: mssparkutils.fs.exists("/tmp/my-file") ## End(Not run)## Not run: mssparkutils.fs.exists("/tmp/my-file") ## End(Not run)
Copies a file or directory via azcopy, possibly across FileSystems.
mssparkutils.fs.fastcp(from, to, recurse = TRUE, extraConfigs = NULL)mssparkutils.fs.fastcp(from, to, recurse = TRUE, extraConfigs = NULL)
from |
FileSystem URI of the source file or directory |
to |
FileSystem URI of the destination file or directory |
recurse |
if TRUE, all files and directories will be recursively copied |
extraConfigs |
extra configs for azcopy, includes flags, timeout, aadToken, sourceLinkedService, destinationLinkedService |
TRUE if all files were successfully copied
## Not run: mssparkutils.fs.fastcp("file:/tmp/my-folder/a", "adls://xxx/tmp/b") ## End(Not run)## Not run: mssparkutils.fs.fastcp("file:/tmp/my-folder/a", "adls://xxx/tmp/b") ## End(Not run)
Gets the local path of the mount point.
mssparkutils.fs.getMountPath(mountPoint, scope = "")mssparkutils.fs.getMountPath(mountPoint, scope = "")
mountPoint |
The directory that was previously mounted. |
scope |
Mount point level, job or workspace, default is job. |
Empty string to mimic the local mounted path related to mountPoint.
mssparkutils.fs.getMountPath("/mnt") mssparkutils.fs.getMountPath("/mnt", " job")mssparkutils.fs.getMountPath("/mnt") mssparkutils.fs.getMountPath("/mnt", " job")
Returns up to the first 'maxBytes' bytes of the given file as a String encoded in UTF-8.
mssparkutils.fs.head(file, maxBytes = 65535)mssparkutils.fs.head(file, maxBytes = 65535)
file |
FileSystem URI |
maxBytes |
Maximum number of bytes to read |
Empty string to mimic the returned content of file.
mssparkutils.fs.head("/tmp/my-folder/my-file") mssparkutils.fs.head("/tmp/my-folder/my-file", 1000)mssparkutils.fs.head("/tmp/my-folder/my-file") mssparkutils.fs.head("/tmp/my-folder/my-file", 1000)
Below is overview about the available methods:
mssparkutils.fs.help(methodName = "")mssparkutils.fs.help(methodName = "")
methodName |
method name to get more information. |
mssparkutils.fs.cp: Copies a file or directory, possibly across FileSystems mssparkutils.fs.mv: Moves a file or directory, possibly across FileSystems mssparkutils.fs.ls: Array -> Lists the contents of a directory mssparkutils.fs.mkdirs: Creates the given directory if it does not exist, also creating any necessary parent directories mssparkutils.fs.put: Writes the given String out to a file, encoded in UTF-8 mssparkutils.fs.head: Returns up to the first 'maxBytes' bytes of the given file as a String encoded in UTF-8 mssparkutils.fs.append: Append the content to a file mssparkutils.fs.rm: Removes a file or directory
No return value, print empty string to mimic the behavior of help method of mssparkutils fs module when used in azure synapse runtime.
mssparkutils.fs.help() mssparkutils.fs.help("ls")mssparkutils.fs.help() mssparkutils.fs.help("ls")
Lists the contents of a directory.
mssparkutils.fs.ls(dir)mssparkutils.fs.ls(dir)
dir |
FileSystem URI |
Empty list to mimic the file list under dir.
mssparkutils.fs.ls("/tmp/my-folder/")mssparkutils.fs.ls("/tmp/my-folder/")
Creates the given directory if it does not exist, also creating any necessary parent * directories.
mssparkutils.fs.mkdirs(dir)mssparkutils.fs.mkdirs(dir)
dir |
FileSystem URI |
FALSE to mimic the result if dir creation fail.
mssparkutils.fs.mkdirs("/tmp/a/b/c")mssparkutils.fs.mkdirs("/tmp/a/b/c")
Attach remote storage (Blob, Gen2, Azure File Share) to all working nodes (driver node and worker nodes)
mssparkutils.fs.mount(source, mountPoint, extraConfigs = NULL)mssparkutils.fs.mount(source, mountPoint, extraConfigs = NULL)
source |
FileSystem URI that contains the source data. |
mountPoint |
The directory of remote source to mount the source. |
extraConfigs |
Extra configurations. |
FALSE to mimic the result if mountPoint creation fail.
mssparkutils.fs.mount("abfss://xxx.dfs.core.windows.net", "/mnt")mssparkutils.fs.mount("abfss://xxx.dfs.core.windows.net", "/mnt")
Show information about what is mounted. Any credentials used to mount the mount points listed will not be displayed.
mssparkutils.fs.mounts(extraConfigs = NULL)mssparkutils.fs.mounts(extraConfigs = NULL)
extraConfigs |
Extra configurations. |
The list of MountPointInfo.
Attach remote storage (Blob, Gen2, Azure File Share) to driver node
mssparkutils.fs.mountToDriverNode(source, mountPoint, extraConfigs = NULL)mssparkutils.fs.mountToDriverNode(source, mountPoint, extraConfigs = NULL)
source |
FileSystem URI that contains the source data. |
mountPoint |
The directory of remote source to mount the source. |
extraConfigs |
Extra configurations. |
TRUE if the path was successfully mounted.
Moves a file or directory, possibly across FileSystems. For intra-FileSystem, it is implemented by hadoop fs rename operation. For inter-FileSystem, This is implemented as a copy followed by delete.
mssparkutils.fs.mv(from, to, createPath = FALSE, overwrite = FALSE)mssparkutils.fs.mv(from, to, createPath = FALSE, overwrite = FALSE)
from |
FileSystem URI of the source file or directory. |
to |
FileSystem URI of the destination file or directory. |
createPath |
if TRUE, will firstly create the parent dir if not exists before move op. |
overwrite |
if TRUE, will overwrite the destination folder if exists. |
FALSE to mimic the result of mv operation fail.
mssparkutils.fs.mv("/tmp/my-folder/", "adls:/xxx/tmp/b")mssparkutils.fs.mv("/tmp/my-folder/", "adls:/xxx/tmp/b")
Writes the given String out to a file, encoded in UTF-8.
mssparkutils.fs.put(file, content, overwrite = FALSE)mssparkutils.fs.put(file, content, overwrite = FALSE)
file |
FileSystem URI. |
content |
Content of file to write, encoded in System default charset. |
overwrite |
If set to TRUE, the file will be overwritten if it existed already. Note that if overwrite is TRUE and the the write fails, the original file. may still be deleted. |
FALSE to mimic the result of file put operation fail.
mssparkutils.fs.put("/tmp/my-file", "Hello world!", TRUE)mssparkutils.fs.put("/tmp/my-file", "Hello world!", TRUE)
Refresh workspace level mount points.
mssparkutils.fs.refreshMounts()mssparkutils.fs.refreshMounts()
FALSE to mimic the refreshMounts fail to refresh mount info.
mssparkutils.fs.refreshMounts()mssparkutils.fs.refreshMounts()
Removes a file or directory.
mssparkutils.fs.rm(dir, recurse = FALSE)mssparkutils.fs.rm(dir, recurse = FALSE)
dir |
FileSystem URI for a single file or a directory. |
recurse |
if TRUE, all files and directories will be recursively deleted. |
FALSE to mimic the result of dir deletion fail.
mssparkutils.fs.rm("/tmp/my-folder/", TRUE)mssparkutils.fs.rm("/tmp/my-folder/", TRUE)
Removes a mount point.
mssparkutils.fs.unmount(mountPoint)mssparkutils.fs.unmount(mountPoint)
mountPoint |
The directory that was previously mounted. |
FALSE to mimic the result of unmount mountPoint fail.
mssparkutils.fs.unmount("/mnt")mssparkutils.fs.unmount("/mnt")
Removes a mount point from driver node.
mssparkutils.fs.unmountFromDriverNode(mountPoint)mssparkutils.fs.unmountFromDriverNode(mountPoint)
mountPoint |
The directory that was previously mounted. |
TRUE if the mount point was successfully unmounted.
Get help message for this module.
mssparkutils.help(methodName = "")mssparkutils.help(methodName = "")
methodName |
method name to get more information. |
No return value, print empty string to mimic the behavior of help method of mssparkutils module when used in azure synapse runtime.
mssparkutils.help()mssparkutils.help()
Create a lakehouse
mssparkutils.lakehouse.create( name, description = "", definition = "", workspaceId = "" )mssparkutils.lakehouse.create( name, description = "", definition = "", workspaceId = "" )
name |
Name of the lakehouse |
description |
Description of the lakehouse |
definition |
Definition of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
A lakehouse object
Delete a lakehouse
mssparkutils.lakehouse.delete(name, workspaceId = "")mssparkutils.lakehouse.delete(name, workspaceId = "")
name |
Name of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
Get a lakehouse
mssparkutils.lakehouse.get(name = "", workspaceId = "")mssparkutils.lakehouse.get(name = "", workspaceId = "")
name |
Name of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
A lakehouse object
mssparkutils.lakehouse.create(name: String, description: String, workspaceId: String): Lakehouse -> Create a lakehouse mssparkutils.lakehouse.get(name: String, workspaceId: String): Lakehouse -> Get a lakehouse mssparkutils.lakehouse.delete(name: String, workspaceId: String): void -> Delete a lakehouse mssparkutils.lakehouse.update(name: String, newName: String, description: String, workspaceId: String): Lakehouse -> Update a lakehouse
mssparkutils.lakehouse.help(methodName = "")mssparkutils.lakehouse.help(methodName = "")
methodName |
method name to get more information |
List all lakehouses
mssparkutils.lakehouse.list(workspaceId = "", maxResults = 1000L)mssparkutils.lakehouse.list(workspaceId = "", maxResults = 1000L)
workspaceId |
Workspace id of the lakehouse, default to current workspace |
maxResults |
Maximum number of lakehouses to return, default to 1000 |
A list of lakehouse objects
Update a lakehouse
mssparkutils.lakehouse.update( name, newName, description = "", workspaceId = "" )mssparkutils.lakehouse.update( name, newName, description = "", workspaceId = "" )
name |
Name of the lakehouse |
newName |
New name of the lakehouse |
description |
Description of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
A lakehouse object
Gets the local path of the notebook resource.
mssparkutils.nbResPath()mssparkutils.nbResPath()
The local path of the notebook resource.
This method lets you exit a notebook with a value.
mssparkutils.notebook.exit(value)mssparkutils.notebook.exit(value)
value |
the value to return when exiting. |
No return value, mimic behavior to set the notebook run exit value using value.
mssparkutils.notebook.exit('exitVal')mssparkutils.notebook.exit('exitVal')
The notebook module.
mssparkutils.notebook.help(methodName = "")mssparkutils.notebook.help(methodName = "")
methodName |
method name to get more information. |
No return value, print empty string to mimic the behavior of help method of mssparkutils notebook module when used in azure synapse runtime.
mssparkutils.notebook.help() mssparkutils.notebook.help("run")mssparkutils.notebook.help() mssparkutils.notebook.help("run")
Runs a notebook and returns its exit value. The notebook will run in the current livy session context by default.
mssparkutils.notebook.run( path, timeoutSeconds = 90, arguments = NULL, workspace = "" )mssparkutils.notebook.run( path, timeoutSeconds = 90, arguments = NULL, workspace = "" )
path |
absolute path to the notebook, e.g. /path/to/notebook. |
timeoutSeconds |
timeout in seconds for the called notebook. |
arguments |
string map of arguments to pass to the notebook. |
workspace |
Workspace of the notebook. |
Empty string to mimic the exitVal set by mssparkutils.notebook.exit.
mssparkutils.notebook.run('NB1') mssparkutils.notebook.run('NB1', 200) mssparkutils.notebook.run('NB1', 200, list("input"=30))mssparkutils.notebook.run('NB1') mssparkutils.notebook.run('NB1', 200) mssparkutils.notebook.run('NB1', 200, list("input"=30))
Runs multiple notebooks concurrently with support for dependency relationships. Details can be found in mssparkutils.notebook.help("runMultiple").
mssparkutils.notebook.runMultiple(pathsOrPipeline)mssparkutils.notebook.runMultiple(pathsOrPipeline)
pathsOrPipeline |
A list of notebook names or a complex data structure (JSON string) that meets the requirements of the com.microsoft.spark.notebook.msutils.impl.MsNotebookPipeline scala class. |
a list of exit values and exceptions for each notebook
provide a way to make people can update the endpoint
mssparkutils.notebook.updateNBSEndpoint(endpoint)mssparkutils.notebook.updateNBSEndpoint(endpoint)
endpoint |
the new point |
Check if the DAG is correctly defined, if the syntax is correct, and if notebooks are found in the workspace without actual execution. Details can be found in mssparkutils.notebook.help("validateDAG").
mssparkutils.notebook.validateDAG(dag)mssparkutils.notebook.validateDAG(dag)
dag |
A list of notebook names or a complex data structure (JSON string) that meets the requirements of the com.microsoft.spark.notebook.msutils.impl.MsNotebookPipeline scala class. |
check result
Get runtime properties
mssparkutils.runtime.context()mssparkutils.runtime.context()
A dummy env object to mimic the result of runtime context method when used in azure synapse runtime.
mssparkutils.runtime.context()mssparkutils.runtime.context()
Set runtime high concurrency mode repl id
mssparkutils.runtime.setHcReplId(replId)mssparkutils.runtime.setHcReplId(replId)
replId |
High concurrency mode repl id |
Stop an interactive session
mssparkutils.session.stop(detach = TRUE)mssparkutils.session.stop(detach = TRUE)
detach |
If detach is True, stop session from standard session, or detach current notebook from high concurrency session; if detach is False, stop session in any session. Default is TRUE. |
Dummy in-memory implementation for local development.
notebookutils.conf.get(key, default = "")notebookutils.conf.get(key, default = "")
key |
Config key |
default |
Default value when key is missing |
Dummy in-memory implementation for local development.
notebookutils.conf.set(key, value)notebookutils.conf.set(key, value)
key |
Config key |
value |
Config value |
Dummy implementation for local development.
notebookutils.connections.getCredential(connectionId, artifactId = "")notebookutils.connections.getCredential(connectionId, artifactId = "")
connectionId |
Connection id |
artifactId |
Artifact id |
A dummy credential object (list)
Get help string for a method.
notebookutils.connections.getHelpString(funcName = "", namespace = "")notebookutils.connections.getHelpString(funcName = "", namespace = "")
funcName |
function name |
namespace |
namespace |
Take linked service name as input and return connection string or credentials depending on the configuration of the linked service.
notebookutils.credentials.getConnectionStringOrCreds(linkedService)notebookutils.credentials.getConnectionStringOrCreds(linkedService)
linkedService |
Linked service name. |
A empty string used to mimic credentials returned by azure synapse runtime for linkedService.
notebookutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')notebookutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')
Take linked service name as input and return full connection string with credentials.
notebookutils.credentials.getFullConnectionString(linkedService)notebookutils.credentials.getFullConnectionString(linkedService)
linkedService |
Linked service name. |
A empty string used to mimic connection string returned by azure synapse runtime for linkedService.
notebookutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')notebookutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')
Return all the properties of a given linked service in string format.
notebookutils.credentials.getPropertiesAll(linkedService)notebookutils.credentials.getPropertiesAll(linkedService)
linkedService |
Linked service name. |
A empty string used to mimic properties string returned by azure synapse runtime for linkedService.
notebookutils.credentials.getPropertiesAll('AzureDataLakeStorage1')notebookutils.credentials.getPropertiesAll('AzureDataLakeStorage1')
Return AKV secret.
notebookutils.credentials.getSecret(akvName, secret, linkedService = NULL)notebookutils.credentials.getSecret(akvName, secret, linkedService = NULL)
akvName |
Azure Key Vault name. |
secret |
name of the secret being fetched. |
linkedService |
linkedService name of the AKV linked service. |
A empty string used to mimic secret returned by azure synapse runtime for given akvName and secret.
notebookutils.credentials.getSecret('akvName', 'secretName') notebookutils.credentials.getSecret('akvName', 'secretName', 'AzureDataLakeStorage1')notebookutils.credentials.getSecret('akvName', 'secretName') notebookutils.credentials.getSecret('akvName', 'secretName', 'AzureDataLakeStorage1')
Return AKV secret using linkedService.
notebookutils.credentials.getSecretWithLS(linkedService, secret)notebookutils.credentials.getSecretWithLS(linkedService, secret)
linkedService |
linkedService name of the AKV linked service. |
secret |
name of the secret being fetched. |
A empty string used to mimic secret returned by azure synapse runtime for given linkedService and secret.
notebookutils.credentials.getSecretWithLS('AzureDataLakeStorage1', 'secretName')notebookutils.credentials.getSecretWithLS('AzureDataLakeStorage1', 'secretName')
Get AAD token for a resource.
notebookutils.credentials.getToken(audience, name = "")notebookutils.credentials.getToken(audience, name = "")
audience |
token audience. |
name |
token audience. |
A empty string used to mimic token returned by azure synapse runtime for accessing resource audience.
notebookutils.credentials.getToken('synapse') notebookutils.credentials.getToken('storage') notebookutils.credentials.getToken('storage', 'storage')notebookutils.credentials.getToken('synapse') notebookutils.credentials.getToken('storage') notebookutils.credentials.getToken('storage', 'storage')
Get help message.
notebookutils.credentials.help()notebookutils.credentials.help()
No return value, print empty string to mimic the behavior of help method of mssparkutils credentials module when used in azure synapse runtime.
notebookutils.credentials.help()notebookutils.credentials.help()
Returns true if the input token is valid (i.e, hasn't expired).
notebookutils.credentials.isValidToken(token)notebookutils.credentials.isValidToken(token)
token |
token to validate. |
FALSE to mimic the result if token is invalid.
notebookutils.credentials.isValidToken('dummyToken')notebookutils.credentials.isValidToken('dummyToken')
Put AKV secret using with or without linkedService.
notebookutils.credentials.putSecret( akvName, secretName, secretValue, linkedService = NULL )notebookutils.credentials.putSecret( akvName, secretName, secretValue, linkedService = NULL )
akvName |
Azure Key Vault name. |
secretName |
name of the secret being written. |
secretValue |
value of the secret being written. |
linkedService |
name of the AKV linked service. |
The secretValue been written.
notebookutils.credentials.putSecret('akvName', 'secretName', 'secretValue') notebookutils.credentials.putSecret('akvName', 'secretName', 'secretValue', 'AzureDataLakeStorage1')notebookutils.credentials.putSecret('akvName', 'secretName', 'secretValue') notebookutils.credentials.putSecret('akvName', 'secretName', 'secretValue', 'AzureDataLakeStorage1')
Put AKV secret using linkedService.
notebookutils.credentials.putSecretWithLS( linkedService, secretName, secretValue )notebookutils.credentials.putSecretWithLS( linkedService, secretName, secretValue )
linkedService |
name of AKV linked service. |
secretName |
name of the secret being written. |
secretValue |
value of the secret being written. |
The secretValue been written.
notebookutils.credentials.putSecretWithLS('AzureDataLakeStorage1', 'secretName', 'secretValue')notebookutils.credentials.putSecretWithLS('AzureDataLakeStorage1', 'secretName', 'secretValue')
Get cluster id.
notebookutils.env.getClusterId()notebookutils.env.getClusterId()
A empty string used to mimic cluster id of azure synapse runtime.
notebookutils.env.getClusterId()notebookutils.env.getClusterId()
Get job Id.
notebookutils.env.getJobId()notebookutils.env.getJobId()
A empty string used to mimic the id of spark job been submitted to azure synapse runtime.
notebookutils.env.getJobId()notebookutils.env.getJobId()
Get pool name.
notebookutils.env.getPoolName()notebookutils.env.getPoolName()
A empty string used to mimic the name of user's azure synapse spark pool.
notebookutils.env.getPoolName()notebookutils.env.getPoolName()
Get user Id.
notebookutils.env.getUserId()notebookutils.env.getUserId()
A empty string used to mimic the id of user.
notebookutils.env.getUserId()notebookutils.env.getUserId()
Get user name.
notebookutils.env.getUserName()notebookutils.env.getUserName()
A empty string used to mimic the name of user.
notebookutils.env.getUserName()notebookutils.env.getUserName()
Get workspace name.
notebookutils.env.getWorkspaceName()notebookutils.env.getWorkspaceName()
A empty string used to mimic the id of the user's azure synapse workspace.
notebookutils.env.getWorkspaceName()notebookutils.env.getWorkspaceName()
Get help message.
notebookutils.env.help()notebookutils.env.help()
No return value, print empty string to mimic the behavior of help method of mssparkutils env module when used in azure synapse runtime.
notebookutils.env.help()notebookutils.env.help()
Send a DELETE request to Fabric.
notebookutils.fabricClient.delete(path, headers = list())notebookutils.fabricClient.delete(path, headers = list())
path |
Path of the request |
headers |
Headers of the request |
Send a GET request to Fabric.
notebookutils.fabricClient.get(path, headers = list())notebookutils.fabricClient.get(path, headers = list())
path |
Path of the request |
headers |
Headers of the request |
RestResponse Response of the request
Get help string for a method.
notebookutils.fabricClient.help(methodName = "")notebookutils.fabricClient.help(methodName = "")
methodName |
Name of the method |
List all capacities in the workspace.
notebookutils.fabricClient.listCapacities(maxResults = 1000L)notebookutils.fabricClient.listCapacities(maxResults = 1000L)
maxResults |
Maximum number of capacities to return, default is 1000 |
Array of Capacity objects
Send a PATCH request to Fabric.
notebookutils.fabricClient.patch(path, content, headers = list())notebookutils.fabricClient.patch(path, content, headers = list())
path |
Path of the request |
content |
Content of the request |
headers |
Headers of the request |
RestResponse Response of the request
Send a POST request to Fabric.
notebookutils.fabricClient.post(path, content, headers = list())notebookutils.fabricClient.post(path, content, headers = list())
path |
Path of the request |
content |
Content of the request |
headers |
Headers of the request |
RestResponse Response of the request
Send a PUT request to Fabric.
notebookutils.fabricClient.put(path, content, headers = list())notebookutils.fabricClient.put(path, content, headers = list())
path |
Path of the request |
content |
Content of the request |
headers |
Headers of the request |
RestResponse Response of the request
Append the given String to a file, encoded in UTF-8.
notebookutils.fs.append(file, content, createFileIfNotExists = FALSE)notebookutils.fs.append(file, content, createFileIfNotExists = FALSE)
file |
FileSystem URI |
content |
Content needs to be append to file, encoded in System default charset. |
createFileIfNotExists |
If set to true, will firstly try to create file if not exists. |
FALSE to mimic the result if file content append fail.
notebookutils.fs.append("/tmp/my-file", "Hello world!") notebookutils.fs.append("/tmp/my-file", "Hello world!", TRUE)notebookutils.fs.append("/tmp/my-file", "Hello world!") notebookutils.fs.append("/tmp/my-file", "Hello world!", TRUE)
Copies a file or directory, possibly across FileSystems.
notebookutils.fs.cp(from, to, recurse = FALSE)notebookutils.fs.cp(from, to, recurse = FALSE)
from |
FileSystem URI of the source file or directory |
to |
FileSystem URI of the destination file or directory |
recurse |
if TRUE, all files and directories will be recursively copied |
FALSE to mimic the result if file or directory from fail to copy to to.
notebookutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b") notebookutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b", TRUE)notebookutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b") notebookutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b", TRUE)
Check if a file or directory exists.
notebookutils.fs.exists(file)notebookutils.fs.exists(file)
file |
FileSystem URI |
TRUE if the file or directory exists
## Not run: notebookutils.fs.exists("/tmp/my-file") ## End(Not run)## Not run: notebookutils.fs.exists("/tmp/my-file") ## End(Not run)
Copies a file or directory via azcopy, possibly across FileSystems.
notebookutils.fs.fastcp(from, to, recurse = TRUE, extraConfigs = NULL)notebookutils.fs.fastcp(from, to, recurse = TRUE, extraConfigs = NULL)
from |
FileSystem URI of the source file or directory |
to |
FileSystem URI of the destination file or directory |
recurse |
if TRUE, all files and directories will be recursively copied |
extraConfigs |
extra configs for azcopy, includes flags, timeout, aadToken, sourceLinkedService, destinationLinkedService |
TRUE if all files were successfully copied
## Not run: notebookutils.fs.fastcp("file:/tmp/my-folder/a", "adls://xxx/tmp/b") ## End(Not run)## Not run: notebookutils.fs.fastcp("file:/tmp/my-folder/a", "adls://xxx/tmp/b") ## End(Not run)
Gets the local path of the mount point.
notebookutils.fs.getMountPath(mountPoint, scope = "")notebookutils.fs.getMountPath(mountPoint, scope = "")
mountPoint |
The directory that was previously mounted. |
scope |
Mount point level, job or workspace, default is job. |
Empty string to mimic the local mounted path related to mountPoint.
notebookutils.fs.getMountPath("/mnt") notebookutils.fs.getMountPath("/mnt", "job")notebookutils.fs.getMountPath("/mnt") notebookutils.fs.getMountPath("/mnt", "job")
Returns up to the first 'maxBytes' bytes of the given file as a String encoded in UTF-8.
notebookutils.fs.head(file, maxBytes = 65535)notebookutils.fs.head(file, maxBytes = 65535)
file |
FileSystem URI |
maxBytes |
Maximum number of bytes to read |
Empty string to mimic the returned content of file.
notebookutils.fs.head("/tmp/my-folder/my-file") notebookutils.fs.head("/tmp/my-folder/my-file", 1000)notebookutils.fs.head("/tmp/my-folder/my-file") notebookutils.fs.head("/tmp/my-folder/my-file", 1000)
Below is overview about the available methods:
notebookutils.fs.help(methodName = "")notebookutils.fs.help(methodName = "")
methodName |
method name to get more information. |
notebookutils.fs.cp: Copies a file or directory, possibly across FileSystems notebookutils.fs.mv: Moves a file or directory, possibly across FileSystems notebookutils.fs.ls: Array -> Lists the contents of a directory notebookutils.fs.mkdirs: Creates the given directory if it does not exist, also creating any necessary parent directories notebookutils.fs.put: Writes the given String out to a file, encoded in UTF-8 notebookutils.fs.head: Returns up to the first 'maxBytes' bytes of the given file as a String encoded in UTF-8 notebookutils.fs.append: Append the content to a file notebookutils.fs.rm: Removes a file or directory
No return value, print empty string to mimic the behavior of help method of mssparkutils fs module when used in azure synapse runtime.
notebookutils.fs.help() notebookutils.fs.help("ls")notebookutils.fs.help() notebookutils.fs.help("ls")
Lists the contents of a directory.
notebookutils.fs.ls(dir)notebookutils.fs.ls(dir)
dir |
FileSystem URI |
Empty list to mimic the file list under dir.
notebookutils.fs.ls("/tmp/my-folder/")notebookutils.fs.ls("/tmp/my-folder/")
Creates the given directory if it does not exist, also creating any necessary parent * directories.
notebookutils.fs.mkdirs(dir)notebookutils.fs.mkdirs(dir)
dir |
FileSystem URI |
FALSE to mimic the result if dir creation fail.
notebookutils.fs.mkdirs("/tmp/a/b/c")notebookutils.fs.mkdirs("/tmp/a/b/c")
Attach remote storage (Blob, Gen2, Azure File Share) to all working nodes (driver node and worker nodes)
notebookutils.fs.mount(source, mountPoint, extraConfigs = NULL)notebookutils.fs.mount(source, mountPoint, extraConfigs = NULL)
source |
FileSystem URI that contains the source data. |
mountPoint |
The directory of remote source to mount the source. |
extraConfigs |
Extra configurations. |
FALSE to mimic the result if mountPoint creation fail.
notebookutils.fs.mount("abfss://xxx.dfs.core.windows.net", "/mnt")notebookutils.fs.mount("abfss://xxx.dfs.core.windows.net", "/mnt")
Show information about what is mounted. Any credentials used to mount the mount points listed will not be displayed.
notebookutils.fs.mounts(extraConfigs = NULL)notebookutils.fs.mounts(extraConfigs = NULL)
extraConfigs |
Extra configurations. |
The list of MountPointInfo.
Attach remote storage (Blob, Gen2, Azure File Share) to driver node
notebookutils.fs.mountToDriverNode(source, mountPoint, extraConfigs = NULL)notebookutils.fs.mountToDriverNode(source, mountPoint, extraConfigs = NULL)
source |
FileSystem URI that contains the source data. |
mountPoint |
The directory of remote source to mount the source. |
extraConfigs |
Extra configurations. |
TRUE if the path was successfully mounted.
Moves a file or directory, possibly across FileSystems. For intra-FileSystem, it is implemented by hadoop fs rename operation. For inter-FileSystem, This is implemented as a copy followed by delete.
notebookutils.fs.mv(from, to, createPath = FALSE, overwrite = FALSE)notebookutils.fs.mv(from, to, createPath = FALSE, overwrite = FALSE)
from |
FileSystem URI of the source file or directory. |
to |
FileSystem URI of the destination file or directory. |
createPath |
if TRUE, will firstly create the parent dir if not exists before move op. |
overwrite |
if TRUE, will overwrite the destination folder if exists. |
FALSE to mimic the result of mv operation fail.
notebookutils.fs.mv("/tmp/my-folder/", "adls:/xxx/tmp/b")notebookutils.fs.mv("/tmp/my-folder/", "adls:/xxx/tmp/b")
Writes the given String out to a file, encoded in UTF-8.
notebookutils.fs.put(file, content, overwrite = FALSE)notebookutils.fs.put(file, content, overwrite = FALSE)
file |
FileSystem URI. |
content |
Content of file to write, encoded in System default charset. |
overwrite |
If set to TRUE, the file will be overwritten if it existed already. Note that if overwrite is TRUE and the the write fails, the original file. may still be deleted. |
FALSE to mimic the result of file put operation fail.
notebookutils.fs.put("/tmp/my-file", "Hello world!", TRUE)notebookutils.fs.put("/tmp/my-file", "Hello world!", TRUE)
Refresh workspace level mount points.
notebookutils.fs.refreshMounts()notebookutils.fs.refreshMounts()
FALSE to mimic the refreshMounts fail to refresh mount info.
notebookutils.fs.refreshMounts()notebookutils.fs.refreshMounts()
Removes a file or directory.
notebookutils.fs.rm(dir, recurse = FALSE)notebookutils.fs.rm(dir, recurse = FALSE)
dir |
FileSystem URI for a single file or a directory. |
recurse |
if TRUE, all files and directories will be recursively deleted. |
FALSE to mimic the result of dir deletion fail.
notebookutils.fs.rm("/tmp/my-folder/", TRUE)notebookutils.fs.rm("/tmp/my-folder/", TRUE)
Removes a mount point.
notebookutils.fs.unmount(mountPoint)notebookutils.fs.unmount(mountPoint)
mountPoint |
The directory that was previously mounted. |
FALSE to mimic the result of unmount mountPoint fail.
notebookutils.fs.unmount("/mnt")notebookutils.fs.unmount("/mnt")
Removes a mount point from driver node.
notebookutils.fs.unmountFromDriverNode(mountPoint)notebookutils.fs.unmountFromDriverNode(mountPoint)
mountPoint |
The directory that was previously mounted. |
TRUE if the mount point was successfully unmounted.
Get help message for this module.
notebookutils.help(methodName = "")notebookutils.help(methodName = "")
methodName |
method name to get more information. |
No return value, print empty string to mimic the behavior of help method of mssparkutils module when used in azure synapse runtime.
notebookutils.help()notebookutils.help()
Initialize notebookutils, using in pre-run code
notebookutils.initialize()notebookutils.initialize()
Create a lakehouse
notebookutils.lakehouse.create( name, description = "", definition = "", workspaceId = "" )notebookutils.lakehouse.create( name, description = "", definition = "", workspaceId = "" )
name |
Name of the lakehouse |
description |
Description of the lakehouse |
definition |
Definition of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
A lakehouse object
Delete a lakehouse
notebookutils.lakehouse.delete(name, workspaceId = "")notebookutils.lakehouse.delete(name, workspaceId = "")
name |
Name of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
Get a lakehouse
notebookutils.lakehouse.get(name = "", workspaceId = "")notebookutils.lakehouse.get(name = "", workspaceId = "")
name |
Name of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
A lakehouse object
Get the definition of a lakehouse
notebookutils.lakehouse.getDefinition(name, workspaceId = "")notebookutils.lakehouse.getDefinition(name, workspaceId = "")
name |
Name of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
The definition of the lakehouse
Get the info of a Lakehouse with properties.
notebookutils.lakehouse.getWithProperties(name, workspaceId = "")notebookutils.lakehouse.getWithProperties(name, workspaceId = "")
name |
Name of the Lakehouse. |
workspaceId |
Id of the workspace, default to current workspace. |
Artifact object. Please refer to: https://learn.microsoft.com/en-us/rest/api/fabric /articles/item-management/properties/lakehouse-properties
notebookutils.lakehouse.create(name: String, description: String, workspaceId: String): Lakehouse -> Create a lakehouse notebookutils.lakehouse.get(name: String, workspaceId: String): Lakehouse -> Get a lakehouse notebookutils.lakehouse.delete(name: String, workspaceId: String): void -> Delete a lakehouse notebookutils.lakehouse.update(name: String, newName: String, description: String, workspaceId: String): Lakehouse -> Update a lakehouse
notebookutils.lakehouse.help(methodName = "")notebookutils.lakehouse.help(methodName = "")
methodName |
method name to get more information |
List all lakehouses
notebookutils.lakehouse.list(workspaceId = "", maxResults = 1000L)notebookutils.lakehouse.list(workspaceId = "", maxResults = 1000L)
workspaceId |
Workspace id of the lakehouse, default to current workspace |
maxResults |
Maximum number of lakehouses to return, default to 1000 |
A list of lakehouse objects
List all tables in a Lakehouse.
notebookutils.lakehouse.listTables( lakehouse = "", workspaceId = "", maxResults = 1000L )notebookutils.lakehouse.listTables( lakehouse = "", workspaceId = "", maxResults = 1000L )
lakehouse |
Name of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
maxResults |
Maximum number of tables to return, default to 1000 |
A list of table objects
Starts a load table operation.
notebookutils.lakehouse.loadTable( loadOption, table, lakehouse = "", workspaceId = "" )notebookutils.lakehouse.loadTable( loadOption, table, lakehouse = "", workspaceId = "" )
loadOption |
string, loadOption Load options. Please refer to https://learn.microsoft.com/en-us/rest/api/fabric/lakehouse/tables/load-table |
table |
Name of the table |
lakehouse |
Name of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
boolean
Update a lakehouse
notebookutils.lakehouse.update( name, newName, description = "", workspaceId = "" )notebookutils.lakehouse.update( name, newName, description = "", workspaceId = "" )
name |
Name of the lakehouse |
newName |
New name of the lakehouse |
description |
Description of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
A lakehouse object
Get the definition of a lakehouse
notebookutils.lakehouse.updateDefinition(name, definition, workspaceId = "")notebookutils.lakehouse.updateDefinition(name, definition, workspaceId = "")
name |
Name of the lakehouse |
definition |
Definition of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
The definition of the lakehouse
Gets the local path of the notebook resource.
notebookutils.nbResPath()notebookutils.nbResPath()
The local path of the notebook resource.
Create a notebook
notebookutils.notebook.create( name, description = "", content = "", defaultLakehouse = "", defaultLakehouseWorkspace = "", workspaceId = "" )notebookutils.notebook.create( name, description = "", content = "", defaultLakehouse = "", defaultLakehouseWorkspace = "", workspaceId = "" )
name |
Name of the notebook |
description |
Description of the notebook |
content |
Definition of the notebook |
defaultLakehouse |
Default lakehouse of the notebook |
defaultLakehouseWorkspace |
Default lakehouse workspace of the notebook |
workspaceId |
Workspace id of the notebook, default to current workspace |
A notebook object
Delete a notebook
notebookutils.notebook.delete(name, workspaceId = "")notebookutils.notebook.delete(name, workspaceId = "")
name |
Name of the notebook |
workspaceId |
Workspace id of the notebook, default to current workspace |
This method lets you exit a notebook with a value.
notebookutils.notebook.exit(value)notebookutils.notebook.exit(value)
value |
the value to return when exiting. |
No return value, mimic behavior to set the notebook run exit value using value.
notebookutils.notebook.exit('exitVal')notebookutils.notebook.exit('exitVal')
Get a notebook
notebookutils.notebook.get(name, workspaceId = "")notebookutils.notebook.get(name, workspaceId = "")
name |
Name of the notebook |
workspaceId |
Workspace id of the notebook, default to current workspace |
A notebook object
The notebook module.
notebookutils.notebook.help(methodName = "")notebookutils.notebook.help(methodName = "")
methodName |
method name to get more information. |
No return value, print empty string to mimic the behavior of help method of mssparkutils notebook module when used in azure synapse runtime.
notebookutils.notebook.help() notebookutils.notebook.help("run")notebookutils.notebook.help() notebookutils.notebook.help("run")
List all notebooks
notebookutils.notebook.list(workspaceId = "", maxResults = 1000L)notebookutils.notebook.list(workspaceId = "", maxResults = 1000L)
workspaceId |
Workspace id of the notebook, default to current workspace |
maxResults |
Maximum number of notebooks to return, default to 1000 |
A list of notebook objects
Runs a notebook and returns its exit value. The notebook will run in the current livy session context by default.
notebookutils.notebook.run( path, timeoutSeconds = 90, arguments = NULL, workspace = "" )notebookutils.notebook.run( path, timeoutSeconds = 90, arguments = NULL, workspace = "" )
path |
absolute path to the notebook, e.g. /path/to/notebook. |
timeoutSeconds |
timeout in seconds for the called notebook. |
arguments |
string map of arguments to pass to the notebook. |
workspace |
Workspace of the notebook. |
Empty string to mimic the exitVal set by mssparkutils.notebook.exit.
notebookutils.notebook.run('NB1') notebookutils.notebook.run('NB1', 200) notebookutils.notebook.run('NB1', 200, list("input"=30))notebookutils.notebook.run('NB1') notebookutils.notebook.run('NB1', 200) notebookutils.notebook.run('NB1', 200, list("input"=30))
Update a notebook
notebookutils.notebook.update( name, newName, description = "", workspaceId = "" )notebookutils.notebook.update( name, newName, description = "", workspaceId = "" )
name |
Name of the notebook |
newName |
New name of the notebook |
description |
Description of the notebook |
workspaceId |
Workspace id of the notebook, default to current workspace |
A notebook object
Get the definition of a notebook
notebookutils.notebook.updateDefinition( name, content, defaultLakehouse = "", defaultLakehouseWorkspace = "", workspaceId = "", environmentId = "", environmentWorkspaceId = "" )notebookutils.notebook.updateDefinition( name, content, defaultLakehouse = "", defaultLakehouseWorkspace = "", workspaceId = "", environmentId = "", environmentWorkspaceId = "" )
name |
Name of the notebook |
content |
Definition of the notebook |
defaultLakehouse |
Default lakehouse of the notebook |
defaultLakehouseWorkspace |
Default lakehouse workspace of the notebook |
workspaceId |
Workspace id of the notebook, default to current workspace |
environmentId |
Environment id of the notebook, default to current environment |
environmentWorkspaceId |
Environment workspace id of the notebook |
The definition of the notebook
provide a way to make people can update the endpoint
notebookutils.notebook.updateNBSEndpoint(endpoint)notebookutils.notebook.updateNBSEndpoint(endpoint)
endpoint |
the new point |
Check if the DAG is correctly defined, if the syntax is correct, and if notebooks are found in the workspace without actual execution. Details can be found in mssparkutils.notebook.help("validateDAG").
notebookutils.notebook.validateDAG(dag)notebookutils.notebook.validateDAG(dag)
dag |
A list of notebook names or a complex data structure (JSON string) that meets the requirements of the com.microsoft.spark.notebook.msutils.impl.MsNotebookPipeline scala class. |
check result
Get runtime properties
notebookutils.runtime.context()notebookutils.runtime.context()
A dummy env object to mimic the result of runtime context method when used in azure synapse runtime.
notebookutils.runtime.context()notebookutils.runtime.context()
notebookutils.runtime is a utility to manage runtime context. context() returns the runtime context as a list.
notebookutils.runtime.help(methodName = "")notebookutils.runtime.help(methodName = "")
methodName |
method name to get more information.am |
Set runtime high concurrency mode repl id
notebookutils.runtime.setHcReplId(replId)notebookutils.runtime.setHcReplId(replId)
replId |
High concurrency mode repl id |
Stop an interactive session
notebookutils.session.stop(detach = TRUE)notebookutils.session.stop(detach = TRUE)
detach |
If detach is True, stop session from standard session, or detach current notebook from high concurrency session; if detach is False, stop session in any session. Default is TRUE. |
Dummy implementation for local development.
notebookutils.udf.getFunctions(udf, workspaceId = "")notebookutils.udf.getFunctions(udf, workspaceId = "")
udf |
UDF artifact id or name |
workspaceId |
Workspace id |
A dummy UDF object (list)
Get help string for a method.
notebookutils.udf.getHelpString(funcName = "", namespace = "")notebookutils.udf.getHelpString(funcName = "", namespace = "")
funcName |
function name |
namespace |
namespace |
Get help string for a method.
notebookutils.udf.help(methodName = "")notebookutils.udf.help(methodName = "")
methodName |
Name of the method |
Run a User data functions (UDF).
notebookutils.udf.run( artifactId, functionName, parameters = list(), workspaceId = "", capacityId = "" )notebookutils.udf.run( artifactId, functionName, parameters = list(), workspaceId = "", capacityId = "" )
artifactId |
The UDF artifact id. |
functionName |
The UDF function name. |
parameters |
The UDF parameters. |
workspaceId |
The UDF workspace id, if not provided, it will be retrieved from the current workspace. |
capacityId |
The UDF capacity id, if not provided, it will be retrieved from the current capacity. |
The UDF execution result.
Dummy implementation for local development.
notebookutils.variableLibrary.get(variableReference)notebookutils.variableLibrary.get(variableReference)
variableReference |
Variable reference |
A variable value (NULL)
Get help string for a method.
notebookutils.variableLibrary.getHelpString(funcName = "", namespace = "")notebookutils.variableLibrary.getHelpString(funcName = "", namespace = "")
funcName |
function name |
namespace |
namespace |
Dummy implementation for local development.
notebookutils.variableLibrary.getLibrary(variableLibraryName)notebookutils.variableLibrary.getLibrary(variableLibraryName)
variableLibraryName |
Variable library name |
A dummy variable library object (list)