| Title: | Dummy R APIs Used in 'Azure Synapse Analytics' for Local Developments |
|---|---|
| Description: | This is a pure dummy interfaces package which mirrors 'MsSparkUtils' APIs <https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/microsoft-spark-utilities?pivots=programming-language-r> of 'Azure Synapse Analytics' <https://learn.microsoft.com/en-us/azure/synapse-analytics/> for R users, customer of Azure Synapse can download this package from CRAN for local development. |
| Authors: | runtimeexp [aut, cre], Microsoft [cph] |
| Maintainer: | runtimeexp <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 1.5.3 |
| Built: | 2024-12-18 06:47:26 UTC |
| Source: | CRAN |
Set the dataframe info which needs to be visualized.
display(dataFrame, isSummary = FALSE)display(dataFrame, isSummary = FALSE)
dataFrame |
the dataframe that needs to be visualized. |
isSummary |
whether show summary info of the dataframe. |
No return value, show the first part of passed dataFrame.
data <- list(56,78,90,45,67) df <- data.frame(t(sapply(data,c))) display(df) display(df, TRUE)data <- list(56,78,90,45,67) df <- data.frame(t(sapply(data,c))) display(df) display(df, TRUE)
display) which needs to be visualized.Set the chart config metadata for current dataframe (set by display) which needs to be visualized.
display.config( commId, lastCommId = NULL, binsNumber = 10, category = "table", keys = NULL, values = NULL, series = NULL, aggregation = NULL, column = NULL )display.config( commId, lastCommId = NULL, binsNumber = 10, category = "table", keys = NULL, values = NULL, series = NULL, aggregation = NULL, column = NULL )
commId |
the id used to identify whether the API call from synapse notebook js client. |
lastCommId |
same with |
binsNumber |
bins number for rendering histogram, default is 10. |
category |
the chart category as |
keys |
the column names which useds to render x-axis. |
values |
the column names which used to render y-axis. |
series |
the column which used to render the chart series |
aggregation |
the aggregation operation type: |
column |
will deperated: the column name used to calculate the statistic info, as the column type, unique values, missing values, etc. |
Construct an specific html fragment to synapse notebook front-end for rendering based on user-input html content.
displayHTML(content)displayHTML(content)
content |
html content which user want to render |
No return value, print the content to mimic the render behavior when used in azure synapse runtime.
displayHTML('<b>Hello world!</b>')displayHTML('<b>Hello world!</b>')
Take linked service name as input and return connection string or credentials depending on the configuration of the linked service.
mssparkutils.credentials.getConnectionStringOrCreds(linkedService)mssparkutils.credentials.getConnectionStringOrCreds(linkedService)
linkedService |
Linked service name. |
A empty string used to mimic credentials returned by azure synapse runtime for linkedService.
mssparkutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')mssparkutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')
Take linked service name as input and return full connection string with credentials.
mssparkutils.credentials.getFullConnectionString(linkedService)mssparkutils.credentials.getFullConnectionString(linkedService)
linkedService |
Linked service name. |
A empty string used to mimic connection string returned by azure synapse runtime for linkedService.
mssparkutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')mssparkutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')
Return all the properties of a given linked service in string format.
mssparkutils.credentials.getPropertiesAll(linkedService)mssparkutils.credentials.getPropertiesAll(linkedService)
linkedService |
Linked service name. |
A empty string used to mimic properties string returned by azure synapse runtime for linkedService.
mssparkutils.credentials.getPropertiesAll('AzureDataLakeStorage1')mssparkutils.credentials.getPropertiesAll('AzureDataLakeStorage1')
Return AKV secret.
mssparkutils.credentials.getSecret(akvName, secret, linkedService = NULL)mssparkutils.credentials.getSecret(akvName, secret, linkedService = NULL)
akvName |
Azure Key Vault name. |
secret |
name of the secret being fetched. |
linkedService |
linkedService name of the AKV linked service. |
A empty string used to mimic secret returned by azure synapse runtime for given akvName and secret.
mssparkutils.credentials.getSecret('akvName', 'secretName') mssparkutils.credentials.getSecret('akvName', 'secretName', 'AzureDataLakeStorage1')mssparkutils.credentials.getSecret('akvName', 'secretName') mssparkutils.credentials.getSecret('akvName', 'secretName', 'AzureDataLakeStorage1')
Return AKV secret using linkedService.
mssparkutils.credentials.getSecretWithLS(linkedService, secret)mssparkutils.credentials.getSecretWithLS(linkedService, secret)
linkedService |
linkedService name of the AKV linked service. |
secret |
name of the secret being fetched. |
A empty string used to mimic secret returned by azure synapse runtime for given linkedService and secret.
mssparkutils.credentials.getSecretWithLS('AzureDataLakeStorage1', 'secretName')mssparkutils.credentials.getSecretWithLS('AzureDataLakeStorage1', 'secretName')
Get AAD token for a resource.
mssparkutils.credentials.getToken(audience, name = "")mssparkutils.credentials.getToken(audience, name = "")
audience |
token audience. |
name |
token audience. |
A empty string used to mimic token returned by azure synapse runtime for accessing resource audience.
mssparkutils.credentials.getToken('synapse') mssparkutils.credentials.getToken('storage') mssparkutils.credentials.getToken('storage', 'storage')mssparkutils.credentials.getToken('synapse') mssparkutils.credentials.getToken('storage') mssparkutils.credentials.getToken('storage', 'storage')
Get help message.
mssparkutils.credentials.help()mssparkutils.credentials.help()
No return value, print empty string to mimic the behavior of help method of mssparkutils credentials module when used in azure synapse runtime.
mssparkutils.credentials.help()mssparkutils.credentials.help()
Returns true if the input token is valid (i.e, hasn't expired).
mssparkutils.credentials.isValidToken(token)mssparkutils.credentials.isValidToken(token)
token |
token to validate. |
FALSE to mimic the result if token is invalid.
mssparkutils.credentials.isValidToken('dummyToken')mssparkutils.credentials.isValidToken('dummyToken')
Put AKV secret using with or without linkedService.
mssparkutils.credentials.putSecret( akvName, secretName, secretValue, linkedService = NULL )mssparkutils.credentials.putSecret( akvName, secretName, secretValue, linkedService = NULL )
akvName |
Azure Key Vault name. |
secretName |
name of the secret being written. |
secretValue |
value of the secret being written. |
linkedService |
name of the AKV linked service. |
The secretValue been written.
mssparkutils.credentials.putSecret('akvName', 'secretName', 'secretValue') mssparkutils.credentials.putSecret('akvName', 'secretName', 'secretValue', 'AzureDataLakeStorage1')mssparkutils.credentials.putSecret('akvName', 'secretName', 'secretValue') mssparkutils.credentials.putSecret('akvName', 'secretName', 'secretValue', 'AzureDataLakeStorage1')
Put AKV secret using linkedService.
mssparkutils.credentials.putSecretWithLS( linkedService, secretName, secretValue )mssparkutils.credentials.putSecretWithLS( linkedService, secretName, secretValue )
linkedService |
name of AKV linked service. |
secretName |
name of the secret being written. |
secretValue |
value of the secret being written. |
The secretValue been written.
mssparkutils.credentials.putSecretWithLS('AzureDataLakeStorage1', 'secretName', 'secretValue')mssparkutils.credentials.putSecretWithLS('AzureDataLakeStorage1', 'secretName', 'secretValue')
Get cluster id.
mssparkutils.env.getClusterId()mssparkutils.env.getClusterId()
A empty string used to mimic cluster id of azure synapse runtime.
mssparkutils.env.getClusterId()mssparkutils.env.getClusterId()
Get job Id.
mssparkutils.env.getJobId()mssparkutils.env.getJobId()
A empty string used to mimic the id of spark job been submitted to azure synapse runtime.
mssparkutils.env.getJobId()mssparkutils.env.getJobId()
Get pool name.
mssparkutils.env.getPoolName()mssparkutils.env.getPoolName()
A empty string used to mimic the name of user's azure synapse spark pool.
mssparkutils.env.getPoolName()mssparkutils.env.getPoolName()
Get user Id.
mssparkutils.env.getUserId()mssparkutils.env.getUserId()
A empty string used to mimic the id of user.
mssparkutils.env.getUserId()mssparkutils.env.getUserId()
Get user name.
mssparkutils.env.getUserName()mssparkutils.env.getUserName()
A empty string used to mimic the name of user.
mssparkutils.env.getUserName()mssparkutils.env.getUserName()
Get workspace name.
mssparkutils.env.getWorkspaceName()mssparkutils.env.getWorkspaceName()
A empty string used to mimic the id of the user's azure synapse workspace.
mssparkutils.env.getWorkspaceName()mssparkutils.env.getWorkspaceName()
Get help message.
mssparkutils.env.help()mssparkutils.env.help()
No return value, print empty string to mimic the behavior of help method of mssparkutils env module when used in azure synapse runtime.
mssparkutils.env.help()mssparkutils.env.help()
Append the given String to a file, encoded in UTF-8.
mssparkutils.fs.append(file, content, createFileIfNotExists = FALSE)mssparkutils.fs.append(file, content, createFileIfNotExists = FALSE)
file |
FileSystem URI |
content |
Content needs to be append to file, encoded in System default charset. |
createFileIfNotExists |
If set to true, will firstly try to create file if not exists. |
FALSE to mimic the result if file content append fail.
mssparkutils.fs.append("/tmp/my-file", "Hello world!") mssparkutils.fs.append("/tmp/my-file", "Hello world!", TRUE)mssparkutils.fs.append("/tmp/my-file", "Hello world!") mssparkutils.fs.append("/tmp/my-file", "Hello world!", TRUE)
Copies a file or directory, possibly across FileSystems.
mssparkutils.fs.cp(from, to, recurse = FALSE)mssparkutils.fs.cp(from, to, recurse = FALSE)
from |
FileSystem URI of the source file or directory |
to |
FileSystem URI of the destination file or directory |
recurse |
if TRUE, all files and directories will be recursively copied |
FALSE to mimic the result if file or directory from fail to copy to to.
mssparkutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b") mssparkutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b", TRUE)mssparkutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b") mssparkutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b", TRUE)
Check if a file or directory exists.
mssparkutils.fs.exists(file)mssparkutils.fs.exists(file)
file |
FileSystem URI |
TRUE if the file or directory exists
## Not run: mssparkutils.fs.exists("/tmp/my-file") ## End(Not run)## Not run: mssparkutils.fs.exists("/tmp/my-file") ## End(Not run)
Copies a file or directory via azcopy, possibly across FileSystems.
mssparkutils.fs.fastcp(from, to, recurse = TRUE, extraConfigs = NULL)mssparkutils.fs.fastcp(from, to, recurse = TRUE, extraConfigs = NULL)
from |
FileSystem URI of the source file or directory |
to |
FileSystem URI of the destination file or directory |
recurse |
if TRUE, all files and directories will be recursively copied |
extraConfigs |
extra configs for azcopy, includes flags, timeout, aadToken, sourceLinkedService, destinationLinkedService |
TRUE if all files were successfully copied
## Not run: mssparkutils.fs.fastcp("file:/tmp/my-folder/a", "adls://xxx/tmp/b") ## End(Not run)## Not run: mssparkutils.fs.fastcp("file:/tmp/my-folder/a", "adls://xxx/tmp/b") ## End(Not run)
Gets the local path of the mount point.
mssparkutils.fs.getMountPath(mountPoint, scope = "job")mssparkutils.fs.getMountPath(mountPoint, scope = "job")
mountPoint |
The directory that was previously mounted. |
scope |
Mount point level, job or workspace, default is job. |
Empty string to mimic the local mounted path related to mountPoint.
mssparkutils.fs.getMountPath("/mnt") mssparkutils.fs.getMountPath("/mnt", "job")mssparkutils.fs.getMountPath("/mnt") mssparkutils.fs.getMountPath("/mnt", "job")
Returns up to the first 'maxBytes' bytes of the given file as a String encoded in UTF-8.
mssparkutils.fs.head(file, maxBytes = 65535)mssparkutils.fs.head(file, maxBytes = 65535)
file |
FileSystem URI |
maxBytes |
Maximum number of bytes to read |
Empty string to mimic the returned content of file.
mssparkutils.fs.head("/tmp/my-folder/my-file") mssparkutils.fs.head("/tmp/my-folder/my-file", 1000)mssparkutils.fs.head("/tmp/my-folder/my-file") mssparkutils.fs.head("/tmp/my-folder/my-file", 1000)
Below is overview about the available methods:
mssparkutils.fs.help(methodName = "")mssparkutils.fs.help(methodName = "")
methodName |
method name to get more information. |
mssparkutils.fs.cp: Copies a file or directory, possibly across FileSystems mssparkutils.fs.mv: Moves a file or directory, possibly across FileSystems mssparkutils.fs.ls: Array -> Lists the contents of a directory mssparkutils.fs.mkdirs: Creates the given directory if it does not exist, also creating any necessary parent directories mssparkutils.fs.put: Writes the given String out to a file, encoded in UTF-8 mssparkutils.fs.head: Returns up to the first 'maxBytes' bytes of the given file as a String encoded in UTF-8 mssparkutils.fs.append: Append the content to a file mssparkutils.fs.rm: Removes a file or directory
No return value, print empty string to mimic the behavior of help method of mssparkutils fs module when used in azure synapse runtime.
mssparkutils.fs.help() mssparkutils.fs.help("ls")mssparkutils.fs.help() mssparkutils.fs.help("ls")
Lists the contents of a directory.
mssparkutils.fs.ls(dir)mssparkutils.fs.ls(dir)
dir |
FileSystem URI |
Empty list to mimic the file list under dir.
mssparkutils.fs.ls("/tmp/my-folder/")mssparkutils.fs.ls("/tmp/my-folder/")
Creates the given directory if it does not exist, also creating any necessary parent * directories.
mssparkutils.fs.mkdirs(dir)mssparkutils.fs.mkdirs(dir)
dir |
FileSystem URI |
FALSE to mimic the result if dir creation fail.
mssparkutils.fs.mkdirs("/tmp/a/b/c")mssparkutils.fs.mkdirs("/tmp/a/b/c")
Attach remote storage (Blob, Gen2, Azure File Share) to all working nodes (driver node and worker nodes)
mssparkutils.fs.mount(source, mountPoint, extraConfigs = NULL)mssparkutils.fs.mount(source, mountPoint, extraConfigs = NULL)
source |
FileSystem URI that contains the source data. |
mountPoint |
The directory of remote source to mount the source. |
extraConfigs |
Extra configurations. |
FALSE to mimic the result if mountPoint creation fail.
mssparkutils.fs.mount("abfss://xxx.dfs.core.windows.net", "/mnt")mssparkutils.fs.mount("abfss://xxx.dfs.core.windows.net", "/mnt")
Show information about what is mounted. Any credentials used to mount the mount points listed will not be displayed.
mssparkutils.fs.mounts(extraConfigs = NULL)mssparkutils.fs.mounts(extraConfigs = NULL)
extraConfigs |
Extra configurations. |
The list of MountPointInfo.
Attach remote storage (Blob, Gen2, Azure File Share) to driver node
mssparkutils.fs.mountToDriverNode(source, mountPoint, extraConfigs = NULL)mssparkutils.fs.mountToDriverNode(source, mountPoint, extraConfigs = NULL)
source |
FileSystem URI that contains the source data. |
mountPoint |
The directory of remote source to mount the source. |
extraConfigs |
Extra configurations. |
TRUE if the path was successfully mounted.
Moves a file or directory, possibly across FileSystems. For intra-FileSystem, it is implemented by hadoop fs rename operation. For inter-FileSystem, This is implemented as a copy followed by delete.
mssparkutils.fs.mv(from, to, createPath = FALSE, overwrite = FALSE)mssparkutils.fs.mv(from, to, createPath = FALSE, overwrite = FALSE)
from |
FileSystem URI of the source file or directory. |
to |
FileSystem URI of the destination file or directory. |
createPath |
if TRUE, will firstly create the parent dir if not exists before move op. |
overwrite |
if TRUE, will overwrite the destination folder if exists. |
FALSE to mimic the result of mv operation fail.
mssparkutils.fs.mv("/tmp/my-folder/", "adls:/xxx/tmp/b")mssparkutils.fs.mv("/tmp/my-folder/", "adls:/xxx/tmp/b")
Writes the given String out to a file, encoded in UTF-8.
mssparkutils.fs.put(file, content, overwrite = FALSE)mssparkutils.fs.put(file, content, overwrite = FALSE)
file |
FileSystem URI. |
content |
Content of file to write, encoded in System default charset. |
overwrite |
If set to TRUE, the file will be overwritten if it existed already. Note that if overwrite is TRUE and the the write fails, the original file. may still be deleted. |
FALSE to mimic the result of file put operation fail.
mssparkutils.fs.put("/tmp/my-file", "Hello world!", TRUE)mssparkutils.fs.put("/tmp/my-file", "Hello world!", TRUE)
Refresh workspace level mount points.
mssparkutils.fs.refreshMounts()mssparkutils.fs.refreshMounts()
FALSE to mimic the refreshMounts fail to refresh mount info.
mssparkutils.fs.refreshMounts()mssparkutils.fs.refreshMounts()
Removes a file or directory.
mssparkutils.fs.rm(dir, recurse = FALSE)mssparkutils.fs.rm(dir, recurse = FALSE)
dir |
FileSystem URI for a single file or a directory. |
recurse |
if TRUE, all files and directories will be recursively deleted. |
FALSE to mimic the result of dir deletion fail.
mssparkutils.fs.rm("/tmp/my-folder/", TRUE)mssparkutils.fs.rm("/tmp/my-folder/", TRUE)
Removes a mount point.
mssparkutils.fs.unmount(mountPoint)mssparkutils.fs.unmount(mountPoint)
mountPoint |
The directory that was previously mounted. |
FALSE to mimic the result of unmount mountPoint fail.
mssparkutils.fs.unmount("/mnt")mssparkutils.fs.unmount("/mnt")
Removes a mount point from driver node.
mssparkutils.fs.unmountFromDriverNode(mountPoint)mssparkutils.fs.unmountFromDriverNode(mountPoint)
mountPoint |
The directory that was previously mounted. |
TRUE if the mount point was successfully unmounted.
Get help message for this module.
mssparkutils.help(methodName = "")mssparkutils.help(methodName = "")
methodName |
method name to get more information. |
No return value, print empty string to mimic the behavior of help method of mssparkutils module when used in azure synapse runtime.
mssparkutils.help()mssparkutils.help()
Create a lakehouse
mssparkutils.lakehouse.create( name, description = "", definition = "", workspaceId = "" )mssparkutils.lakehouse.create( name, description = "", definition = "", workspaceId = "" )
name |
Name of the lakehouse |
description |
Description of the lakehouse |
definition |
Definition of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
A lakehouse object
Delete a lakehouse
mssparkutils.lakehouse.delete(name, workspaceId = "")mssparkutils.lakehouse.delete(name, workspaceId = "")
name |
Name of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
Get a lakehouse
mssparkutils.lakehouse.get(name = "", workspaceId = "")mssparkutils.lakehouse.get(name = "", workspaceId = "")
name |
Name of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
A lakehouse object
mssparkutils.lakehouse.create(name: String, description: String, workspaceId: String): Lakehouse -> Create a lakehouse mssparkutils.lakehouse.get(name: String, workspaceId: String): Lakehouse -> Get a lakehouse mssparkutils.lakehouse.delete(name: String, workspaceId: String): void -> Delete a lakehouse mssparkutils.lakehouse.update(name: String, newName: String, description: String, workspaceId: String): Lakehouse -> Update a lakehouse
mssparkutils.lakehouse.help(methodName = "")mssparkutils.lakehouse.help(methodName = "")
methodName |
method name to get more information |
List all lakehouses
mssparkutils.lakehouse.list(workspaceId = "", maxResults = 1000L)mssparkutils.lakehouse.list(workspaceId = "", maxResults = 1000L)
workspaceId |
Workspace id of the lakehouse, default to current workspace |
maxResults |
Maximum number of lakehouses to return, default to 1000 |
A list of lakehouse objects
Update a lakehouse
mssparkutils.lakehouse.update( name, newName, description = "", workspaceId = "" )mssparkutils.lakehouse.update( name, newName, description = "", workspaceId = "" )
name |
Name of the lakehouse |
newName |
New name of the lakehouse |
description |
Description of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
A lakehouse object
This method lets you exit a notebook with a value.
mssparkutils.notebook.exit(value)mssparkutils.notebook.exit(value)
value |
the value to return when exiting. |
No return value, mimic behavior to set the notebook run exit value using value.
mssparkutils.notebook.exit('exitVal')mssparkutils.notebook.exit('exitVal')
The notebook module.
mssparkutils.notebook.help(methodName = "")mssparkutils.notebook.help(methodName = "")
methodName |
method name to get more information. |
No return value, print empty string to mimic the behavior of help method of mssparkutils notebook module when used in azure synapse runtime.
mssparkutils.notebook.help() mssparkutils.notebook.help("run")mssparkutils.notebook.help() mssparkutils.notebook.help("run")
Runs a notebook and returns its exit value. The notebook will run in the current livy session context by default.
mssparkutils.notebook.run(path, timeoutSeconds = 90, arguments = NULL)mssparkutils.notebook.run(path, timeoutSeconds = 90, arguments = NULL)
path |
absolute path to the notebook, e.g. /path/to/notebook. |
timeoutSeconds |
timeout in seconds for the called notebook. |
arguments |
string map of arguments to pass to the notebook. |
Empty string to mimic the exitVal set by mssparkutils.notebook.exit.
mssparkutils.notebook.run('NB1') mssparkutils.notebook.run('NB1', 200) mssparkutils.notebook.run('NB1', 200, list("input"=30))mssparkutils.notebook.run('NB1') mssparkutils.notebook.run('NB1', 200) mssparkutils.notebook.run('NB1', 200, list("input"=30))
Runs multiple notebooks concurrently with support for dependency relationships. Details can be found in mssparkutils.notebook.help("runMultiple").
mssparkutils.notebook.runMultiple(pathsOrPipeline)mssparkutils.notebook.runMultiple(pathsOrPipeline)
pathsOrPipeline |
A list of notebook names or a complex data structure (JSON string) that meets the requirements of the com.microsoft.spark.notebook.msutils.impl.MsNotebookPipeline scala class. |
a list of exit values and exceptions for each notebook
provide a way to make people can update the endpoint
mssparkutils.notebook.updateNBSEndpoint(endpoint)mssparkutils.notebook.updateNBSEndpoint(endpoint)
endpoint |
the new point |
Get runtime properties
mssparkutils.runtime.context()mssparkutils.runtime.context()
A dummy env object to mimic the result of runtime context method when used in azure synapse runtime.
mssparkutils.runtime.context()mssparkutils.runtime.context()
Set runtime high concurrency mode repl id
mssparkutils.runtime.setHcReplId(replId)mssparkutils.runtime.setHcReplId(replId)
replId |
High concurrency mode repl id |
Stop an interactive session
mssparkutils.session.stop(detach = TRUE)mssparkutils.session.stop(detach = TRUE)
detach |
If detach is True, stop session from standard session, or detach current notebook from high concurrency session; if detach is False, stop session in any session. Default is TRUE. |
Take linked service name as input and return connection string or credentials depending on the configuration of the linked service.
notebookutils.credentials.getConnectionStringOrCreds(linkedService)notebookutils.credentials.getConnectionStringOrCreds(linkedService)
linkedService |
Linked service name. |
A empty string used to mimic credentials returned by azure synapse runtime for linkedService.
notebookutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')notebookutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')
Take linked service name as input and return full connection string with credentials.
notebookutils.credentials.getFullConnectionString(linkedService)notebookutils.credentials.getFullConnectionString(linkedService)
linkedService |
Linked service name. |
A empty string used to mimic connection string returned by azure synapse runtime for linkedService.
notebookutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')notebookutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')
Return all the properties of a given linked service in string format.
notebookutils.credentials.getPropertiesAll(linkedService)notebookutils.credentials.getPropertiesAll(linkedService)
linkedService |
Linked service name. |
A empty string used to mimic properties string returned by azure synapse runtime for linkedService.
notebookutils.credentials.getPropertiesAll('AzureDataLakeStorage1')notebookutils.credentials.getPropertiesAll('AzureDataLakeStorage1')
Return AKV secret.
notebookutils.credentials.getSecret(akvName, secret, linkedService = NULL)notebookutils.credentials.getSecret(akvName, secret, linkedService = NULL)
akvName |
Azure Key Vault name. |
secret |
name of the secret being fetched. |
linkedService |
linkedService name of the AKV linked service. |
A empty string used to mimic secret returned by azure synapse runtime for given akvName and secret.
notebookutils.credentials.getSecret('akvName', 'secretName') notebookutils.credentials.getSecret('akvName', 'secretName', 'AzureDataLakeStorage1')notebookutils.credentials.getSecret('akvName', 'secretName') notebookutils.credentials.getSecret('akvName', 'secretName', 'AzureDataLakeStorage1')
Return AKV secret using linkedService.
notebookutils.credentials.getSecretWithLS(linkedService, secret)notebookutils.credentials.getSecretWithLS(linkedService, secret)
linkedService |
linkedService name of the AKV linked service. |
secret |
name of the secret being fetched. |
A empty string used to mimic secret returned by azure synapse runtime for given linkedService and secret.
notebookutils.credentials.getSecretWithLS('AzureDataLakeStorage1', 'secretName')notebookutils.credentials.getSecretWithLS('AzureDataLakeStorage1', 'secretName')
Get AAD token for a resource.
notebookutils.credentials.getToken(audience, name = "")notebookutils.credentials.getToken(audience, name = "")
audience |
token audience. |
name |
token audience. |
A empty string used to mimic token returned by azure synapse runtime for accessing resource audience.
notebookutils.credentials.getToken('synapse') notebookutils.credentials.getToken('storage') notebookutils.credentials.getToken('storage', 'storage')notebookutils.credentials.getToken('synapse') notebookutils.credentials.getToken('storage') notebookutils.credentials.getToken('storage', 'storage')
Get help message.
notebookutils.credentials.help()notebookutils.credentials.help()
No return value, print empty string to mimic the behavior of help method of mssparkutils credentials module when used in azure synapse runtime.
notebookutils.credentials.help()notebookutils.credentials.help()
Returns true if the input token is valid (i.e, hasn't expired).
notebookutils.credentials.isValidToken(token)notebookutils.credentials.isValidToken(token)
token |
token to validate. |
FALSE to mimic the result if token is invalid.
notebookutils.credentials.isValidToken('dummyToken')notebookutils.credentials.isValidToken('dummyToken')
Put AKV secret using with or without linkedService.
notebookutils.credentials.putSecret( akvName, secretName, secretValue, linkedService = NULL )notebookutils.credentials.putSecret( akvName, secretName, secretValue, linkedService = NULL )
akvName |
Azure Key Vault name. |
secretName |
name of the secret being written. |
secretValue |
value of the secret being written. |
linkedService |
name of the AKV linked service. |
The secretValue been written.
notebookutils.credentials.putSecret('akvName', 'secretName', 'secretValue') notebookutils.credentials.putSecret('akvName', 'secretName', 'secretValue', 'AzureDataLakeStorage1')notebookutils.credentials.putSecret('akvName', 'secretName', 'secretValue') notebookutils.credentials.putSecret('akvName', 'secretName', 'secretValue', 'AzureDataLakeStorage1')
Put AKV secret using linkedService.
notebookutils.credentials.putSecretWithLS( linkedService, secretName, secretValue )notebookutils.credentials.putSecretWithLS( linkedService, secretName, secretValue )
linkedService |
name of AKV linked service. |
secretName |
name of the secret being written. |
secretValue |
value of the secret being written. |
The secretValue been written.
notebookutils.credentials.putSecretWithLS('AzureDataLakeStorage1', 'secretName', 'secretValue')notebookutils.credentials.putSecretWithLS('AzureDataLakeStorage1', 'secretName', 'secretValue')
Get cluster id.
notebookutils.env.getClusterId()notebookutils.env.getClusterId()
A empty string used to mimic cluster id of azure synapse runtime.
notebookutils.env.getClusterId()notebookutils.env.getClusterId()
Get job Id.
notebookutils.env.getJobId()notebookutils.env.getJobId()
A empty string used to mimic the id of spark job been submitted to azure synapse runtime.
notebookutils.env.getJobId()notebookutils.env.getJobId()
Get pool name.
notebookutils.env.getPoolName()notebookutils.env.getPoolName()
A empty string used to mimic the name of user's azure synapse spark pool.
notebookutils.env.getPoolName()notebookutils.env.getPoolName()
Get user Id.
notebookutils.env.getUserId()notebookutils.env.getUserId()
A empty string used to mimic the id of user.
notebookutils.env.getUserId()notebookutils.env.getUserId()
Get user name.
notebookutils.env.getUserName()notebookutils.env.getUserName()
A empty string used to mimic the name of user.
notebookutils.env.getUserName()notebookutils.env.getUserName()
Get workspace name.
notebookutils.env.getWorkspaceName()notebookutils.env.getWorkspaceName()
A empty string used to mimic the id of the user's azure synapse workspace.
notebookutils.env.getWorkspaceName()notebookutils.env.getWorkspaceName()
Get help message.
notebookutils.env.help()notebookutils.env.help()
No return value, print empty string to mimic the behavior of help method of mssparkutils env module when used in azure synapse runtime.
notebookutils.env.help()notebookutils.env.help()
Send a DELETE request to Fabric.
notebookutils.fabricClient.delete(path, headers = list())notebookutils.fabricClient.delete(path, headers = list())
path |
Path of the request |
headers |
Headers of the request |
Send a GET request to Fabric.
notebookutils.fabricClient.get(path, headers = list())notebookutils.fabricClient.get(path, headers = list())
path |
Path of the request |
headers |
Headers of the request |
RestResponse Response of the request
Get help string for a method.
notebookutils.fabricClient.help(methodName = "")notebookutils.fabricClient.help(methodName = "")
methodName |
Name of the method |
List all capacities in the workspace.
notebookutils.fabricClient.listCapacities(maxResults = 1000L)notebookutils.fabricClient.listCapacities(maxResults = 1000L)
maxResults |
Maximum number of capacities to return, default is 1000 |
Array of Capacity objects
Send a PATCH request to Fabric.
notebookutils.fabricClient.patch(path, content, headers = list())notebookutils.fabricClient.patch(path, content, headers = list())
path |
Path of the request |
content |
Content of the request |
headers |
Headers of the request |
RestResponse Response of the request
Send a POST request to Fabric.
notebookutils.fabricClient.post(path, content, headers = list())notebookutils.fabricClient.post(path, content, headers = list())
path |
Path of the request |
content |
Content of the request |
headers |
Headers of the request |
RestResponse Response of the request
Send a PUT request to Fabric.
notebookutils.fabricClient.put(path, content, headers = list())notebookutils.fabricClient.put(path, content, headers = list())
path |
Path of the request |
content |
Content of the request |
headers |
Headers of the request |
RestResponse Response of the request
Append the given String to a file, encoded in UTF-8.
notebookutils.fs.append(file, content, createFileIfNotExists = FALSE)notebookutils.fs.append(file, content, createFileIfNotExists = FALSE)
file |
FileSystem URI |
content |
Content needs to be append to file, encoded in System default charset. |
createFileIfNotExists |
If set to true, will firstly try to create file if not exists. |
FALSE to mimic the result if file content append fail.
notebookutils.fs.append("/tmp/my-file", "Hello world!") notebookutils.fs.append("/tmp/my-file", "Hello world!", TRUE)notebookutils.fs.append("/tmp/my-file", "Hello world!") notebookutils.fs.append("/tmp/my-file", "Hello world!", TRUE)
Copies a file or directory, possibly across FileSystems.
notebookutils.fs.cp(from, to, recurse = FALSE)notebookutils.fs.cp(from, to, recurse = FALSE)
from |
FileSystem URI of the source file or directory |
to |
FileSystem URI of the destination file or directory |
recurse |
if TRUE, all files and directories will be recursively copied |
FALSE to mimic the result if file or directory from fail to copy to to.
notebookutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b") notebookutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b", TRUE)notebookutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b") notebookutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b", TRUE)
Check if a file or directory exists.
notebookutils.fs.exists(file)notebookutils.fs.exists(file)
file |
FileSystem URI |
TRUE if the file or directory exists
## Not run: notebookutils.fs.exists("/tmp/my-file") ## End(Not run)## Not run: notebookutils.fs.exists("/tmp/my-file") ## End(Not run)
Copies a file or directory via azcopy, possibly across FileSystems.
notebookutils.fs.fastcp(from, to, recurse = TRUE, extraConfigs = NULL)notebookutils.fs.fastcp(from, to, recurse = TRUE, extraConfigs = NULL)
from |
FileSystem URI of the source file or directory |
to |
FileSystem URI of the destination file or directory |
recurse |
if TRUE, all files and directories will be recursively copied |
extraConfigs |
extra configs for azcopy, includes flags, timeout, aadToken, sourceLinkedService, destinationLinkedService |
TRUE if all files were successfully copied
## Not run: notebookutils.fs.fastcp("file:/tmp/my-folder/a", "adls://xxx/tmp/b") ## End(Not run)## Not run: notebookutils.fs.fastcp("file:/tmp/my-folder/a", "adls://xxx/tmp/b") ## End(Not run)
Gets the local path of the mount point.
notebookutils.fs.getMountPath(mountPoint, scope = "job")notebookutils.fs.getMountPath(mountPoint, scope = "job")
mountPoint |
The directory that was previously mounted. |
scope |
Mount point level, job or workspace, default is job. |
Empty string to mimic the local mounted path related to mountPoint.
notebookutils.fs.getMountPath("/mnt") notebookutils.fs.getMountPath("/mnt", "job")notebookutils.fs.getMountPath("/mnt") notebookutils.fs.getMountPath("/mnt", "job")
Returns up to the first 'maxBytes' bytes of the given file as a String encoded in UTF-8.
notebookutils.fs.head(file, maxBytes = 65535)notebookutils.fs.head(file, maxBytes = 65535)
file |
FileSystem URI |
maxBytes |
Maximum number of bytes to read |
Empty string to mimic the returned content of file.
notebookutils.fs.head("/tmp/my-folder/my-file") notebookutils.fs.head("/tmp/my-folder/my-file", 1000)notebookutils.fs.head("/tmp/my-folder/my-file") notebookutils.fs.head("/tmp/my-folder/my-file", 1000)
Below is overview about the available methods:
notebookutils.fs.help(methodName = "")notebookutils.fs.help(methodName = "")
methodName |
method name to get more information. |
notebookutils.fs.cp: Copies a file or directory, possibly across FileSystems notebookutils.fs.mv: Moves a file or directory, possibly across FileSystems notebookutils.fs.ls: Array -> Lists the contents of a directory notebookutils.fs.mkdirs: Creates the given directory if it does not exist, also creating any necessary parent directories notebookutils.fs.put: Writes the given String out to a file, encoded in UTF-8 notebookutils.fs.head: Returns up to the first 'maxBytes' bytes of the given file as a String encoded in UTF-8 notebookutils.fs.append: Append the content to a file notebookutils.fs.rm: Removes a file or directory
No return value, print empty string to mimic the behavior of help method of mssparkutils fs module when used in azure synapse runtime.
notebookutils.fs.help() notebookutils.fs.help("ls")notebookutils.fs.help() notebookutils.fs.help("ls")
Lists the contents of a directory.
notebookutils.fs.ls(dir)notebookutils.fs.ls(dir)
dir |
FileSystem URI |
Empty list to mimic the file list under dir.
notebookutils.fs.ls("/tmp/my-folder/")notebookutils.fs.ls("/tmp/my-folder/")
Creates the given directory if it does not exist, also creating any necessary parent * directories.
notebookutils.fs.mkdirs(dir)notebookutils.fs.mkdirs(dir)
dir |
FileSystem URI |
FALSE to mimic the result if dir creation fail.
notebookutils.fs.mkdirs("/tmp/a/b/c")notebookutils.fs.mkdirs("/tmp/a/b/c")
Attach remote storage (Blob, Gen2, Azure File Share) to all working nodes (driver node and worker nodes)
notebookutils.fs.mount(source, mountPoint, extraConfigs = NULL)notebookutils.fs.mount(source, mountPoint, extraConfigs = NULL)
source |
FileSystem URI that contains the source data. |
mountPoint |
The directory of remote source to mount the source. |
extraConfigs |
Extra configurations. |
FALSE to mimic the result if mountPoint creation fail.
notebookutils.fs.mount("abfss://xxx.dfs.core.windows.net", "/mnt")notebookutils.fs.mount("abfss://xxx.dfs.core.windows.net", "/mnt")
Show information about what is mounted. Any credentials used to mount the mount points listed will not be displayed.
notebookutils.fs.mounts(extraConfigs = NULL)notebookutils.fs.mounts(extraConfigs = NULL)
extraConfigs |
Extra configurations. |
The list of MountPointInfo.
Attach remote storage (Blob, Gen2, Azure File Share) to driver node
notebookutils.fs.mountToDriverNode(source, mountPoint, extraConfigs = NULL)notebookutils.fs.mountToDriverNode(source, mountPoint, extraConfigs = NULL)
source |
FileSystem URI that contains the source data. |
mountPoint |
The directory of remote source to mount the source. |
extraConfigs |
Extra configurations. |
TRUE if the path was successfully mounted.
Moves a file or directory, possibly across FileSystems. For intra-FileSystem, it is implemented by hadoop fs rename operation. For inter-FileSystem, This is implemented as a copy followed by delete.
notebookutils.fs.mv(from, to, createPath = FALSE, overwrite = FALSE)notebookutils.fs.mv(from, to, createPath = FALSE, overwrite = FALSE)
from |
FileSystem URI of the source file or directory. |
to |
FileSystem URI of the destination file or directory. |
createPath |
if TRUE, will firstly create the parent dir if not exists before move op. |
overwrite |
if TRUE, will overwrite the destination folder if exists. |
FALSE to mimic the result of mv operation fail.
notebookutils.fs.mv("/tmp/my-folder/", "adls:/xxx/tmp/b")notebookutils.fs.mv("/tmp/my-folder/", "adls:/xxx/tmp/b")
Writes the given String out to a file, encoded in UTF-8.
notebookutils.fs.put(file, content, overwrite = FALSE)notebookutils.fs.put(file, content, overwrite = FALSE)
file |
FileSystem URI. |
content |
Content of file to write, encoded in System default charset. |
overwrite |
If set to TRUE, the file will be overwritten if it existed already. Note that if overwrite is TRUE and the the write fails, the original file. may still be deleted. |
FALSE to mimic the result of file put operation fail.
notebookutils.fs.put("/tmp/my-file", "Hello world!", TRUE)notebookutils.fs.put("/tmp/my-file", "Hello world!", TRUE)
Refresh workspace level mount points.
notebookutils.fs.refreshMounts()notebookutils.fs.refreshMounts()
FALSE to mimic the refreshMounts fail to refresh mount info.
notebookutils.fs.refreshMounts()notebookutils.fs.refreshMounts()
Removes a file or directory.
notebookutils.fs.rm(dir, recurse = FALSE)notebookutils.fs.rm(dir, recurse = FALSE)
dir |
FileSystem URI for a single file or a directory. |
recurse |
if TRUE, all files and directories will be recursively deleted. |
FALSE to mimic the result of dir deletion fail.
notebookutils.fs.rm("/tmp/my-folder/", TRUE)notebookutils.fs.rm("/tmp/my-folder/", TRUE)
Removes a mount point.
notebookutils.fs.unmount(mountPoint)notebookutils.fs.unmount(mountPoint)
mountPoint |
The directory that was previously mounted. |
FALSE to mimic the result of unmount mountPoint fail.
notebookutils.fs.unmount("/mnt")notebookutils.fs.unmount("/mnt")
Removes a mount point from driver node.
notebookutils.fs.unmountFromDriverNode(mountPoint)notebookutils.fs.unmountFromDriverNode(mountPoint)
mountPoint |
The directory that was previously mounted. |
TRUE if the mount point was successfully unmounted.
Get help message for this module.
notebookutils.help(methodName = "")notebookutils.help(methodName = "")
methodName |
method name to get more information. |
No return value, print empty string to mimic the behavior of help method of mssparkutils module when used in azure synapse runtime.
notebookutils.help()notebookutils.help()
Create a lakehouse
notebookutils.lakehouse.create( name, description = "", definition = "", workspaceId = "" )notebookutils.lakehouse.create( name, description = "", definition = "", workspaceId = "" )
name |
Name of the lakehouse |
description |
Description of the lakehouse |
definition |
Definition of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
A lakehouse object
Delete a lakehouse
notebookutils.lakehouse.delete(name, workspaceId = "")notebookutils.lakehouse.delete(name, workspaceId = "")
name |
Name of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
Get a lakehouse
notebookutils.lakehouse.get(name = "", workspaceId = "")notebookutils.lakehouse.get(name = "", workspaceId = "")
name |
Name of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
A lakehouse object
Get the definition of a lakehouse
notebookutils.lakehouse.getDefinition(name, workspaceId = "")notebookutils.lakehouse.getDefinition(name, workspaceId = "")
name |
Name of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
The definition of the lakehouse
Get the info of a Lakehouse with properties.
notebookutils.lakehouse.getWithProperties(name, workspaceId = "")notebookutils.lakehouse.getWithProperties(name, workspaceId = "")
name |
Name of the Lakehouse. |
workspaceId |
Id of the workspace, default to current workspace. |
Artifact object. Please refer to: https://learn.microsoft.com/en-us/rest/api/fabric /articles/item-management/properties/lakehouse-properties
notebookutils.lakehouse.create(name: String, description: String, workspaceId: String): Lakehouse -> Create a lakehouse notebookutils.lakehouse.get(name: String, workspaceId: String): Lakehouse -> Get a lakehouse notebookutils.lakehouse.delete(name: String, workspaceId: String): void -> Delete a lakehouse notebookutils.lakehouse.update(name: String, newName: String, description: String, workspaceId: String): Lakehouse -> Update a lakehouse
notebookutils.lakehouse.help(methodName = "")notebookutils.lakehouse.help(methodName = "")
methodName |
method name to get more information |
List all lakehouses
notebookutils.lakehouse.list(workspaceId = "", maxResults = 1000L)notebookutils.lakehouse.list(workspaceId = "", maxResults = 1000L)
workspaceId |
Workspace id of the lakehouse, default to current workspace |
maxResults |
Maximum number of lakehouses to return, default to 1000 |
A list of lakehouse objects
List all tables in a Lakehouse.
notebookutils.lakehouse.listTables( lakehouse = "", workspaceId = "", maxResults = 1000L )notebookutils.lakehouse.listTables( lakehouse = "", workspaceId = "", maxResults = 1000L )
lakehouse |
Name of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
maxResults |
Maximum number of tables to return, default to 1000 |
A list of table objects
Starts a load table operation.
notebookutils.lakehouse.loadTable( loadOption, table, lakehouse = "", workspaceId = "" )notebookutils.lakehouse.loadTable( loadOption, table, lakehouse = "", workspaceId = "" )
loadOption |
string, loadOption Load options. Please refer to https://learn.microsoft.com/en-us/rest/api/fabric/lakehouse/tables/load-table |
table |
Name of the table |
lakehouse |
Name of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
boolean
Update a lakehouse
notebookutils.lakehouse.update( name, newName, description = "", workspaceId = "" )notebookutils.lakehouse.update( name, newName, description = "", workspaceId = "" )
name |
Name of the lakehouse |
newName |
New name of the lakehouse |
description |
Description of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
A lakehouse object
Get the definition of a lakehouse
notebookutils.lakehouse.updateDefinition(name, definition, workspaceId = "")notebookutils.lakehouse.updateDefinition(name, definition, workspaceId = "")
name |
Name of the lakehouse |
definition |
Definition of the lakehouse |
workspaceId |
Workspace id of the lakehouse, default to current workspace |
The definition of the lakehouse
Create a notebook
notebookutils.notebook.create( name, description = "", content = "", defaultLakehouse = "", defaultLakehouseWorkspace = "", workspaceId = "" )notebookutils.notebook.create( name, description = "", content = "", defaultLakehouse = "", defaultLakehouseWorkspace = "", workspaceId = "" )
name |
Name of the notebook |
description |
Description of the notebook |
content |
Definition of the notebook |
defaultLakehouse |
Default lakehouse of the notebook |
defaultLakehouseWorkspace |
Default lakehouse workspace of the notebook |
workspaceId |
Workspace id of the notebook, default to current workspace |
A notebook object
Delete a notebook
notebookutils.notebook.delete(name, workspaceId = "")notebookutils.notebook.delete(name, workspaceId = "")
name |
Name of the notebook |
workspaceId |
Workspace id of the notebook, default to current workspace |
This method lets you exit a notebook with a value.
notebookutils.notebook.exit(value)notebookutils.notebook.exit(value)
value |
the value to return when exiting. |
No return value, mimic behavior to set the notebook run exit value using value.
notebookutils.notebook.exit('exitVal')notebookutils.notebook.exit('exitVal')
Get a notebook
notebookutils.notebook.get(name, workspaceId = "")notebookutils.notebook.get(name, workspaceId = "")
name |
Name of the notebook |
workspaceId |
Workspace id of the notebook, default to current workspace |
A notebook object
The notebook module.
notebookutils.notebook.help(methodName = "")notebookutils.notebook.help(methodName = "")
methodName |
method name to get more information. |
No return value, print empty string to mimic the behavior of help method of mssparkutils notebook module when used in azure synapse runtime.
notebookutils.notebook.help() notebookutils.notebook.help("run")notebookutils.notebook.help() notebookutils.notebook.help("run")
List all notebooks
notebookutils.notebook.list(workspaceId = "", maxResults = 1000L)notebookutils.notebook.list(workspaceId = "", maxResults = 1000L)
workspaceId |
Workspace id of the notebook, default to current workspace |
maxResults |
Maximum number of notebooks to return, default to 1000 |
A list of notebook objects
Runs a notebook and returns its exit value. The notebook will run in the current livy session context by default.
notebookutils.notebook.run(path, timeoutSeconds = 90, arguments = NULL)notebookutils.notebook.run(path, timeoutSeconds = 90, arguments = NULL)
path |
absolute path to the notebook, e.g. /path/to/notebook. |
timeoutSeconds |
timeout in seconds for the called notebook. |
arguments |
string map of arguments to pass to the notebook. |
Empty string to mimic the exitVal set by mssparkutils.notebook.exit.
notebookutils.notebook.run('NB1') notebookutils.notebook.run('NB1', 200) notebookutils.notebook.run('NB1', 200, list("input"=30))notebookutils.notebook.run('NB1') notebookutils.notebook.run('NB1', 200) notebookutils.notebook.run('NB1', 200, list("input"=30))
Update a notebook
notebookutils.notebook.update( name, newName, description = "", workspaceId = "" )notebookutils.notebook.update( name, newName, description = "", workspaceId = "" )
name |
Name of the notebook |
newName |
New name of the notebook |
description |
Description of the notebook |
workspaceId |
Workspace id of the notebook, default to current workspace |
A notebook object
Get the definition of a notebook
notebookutils.notebook.updateDefinition( name, content, defaultLakehouse = "", defaultLakehouseWorkspace = "", workspaceId = "" )notebookutils.notebook.updateDefinition( name, content, defaultLakehouse = "", defaultLakehouseWorkspace = "", workspaceId = "" )
name |
Name of the notebook |
content |
Definition of the notebook |
defaultLakehouse |
Default lakehouse of the notebook |
defaultLakehouseWorkspace |
Default lakehouse workspace of the notebook |
workspaceId |
Workspace id of the notebook, default to current workspace |
The definition of the notebook
provide a way to make people can update the endpoint
notebookutils.notebook.updateNBSEndpoint(endpoint)notebookutils.notebook.updateNBSEndpoint(endpoint)
endpoint |
the new point |
Get runtime properties
notebookutils.runtime.context()notebookutils.runtime.context()
A dummy env object to mimic the result of runtime context method when used in azure synapse runtime.
notebookutils.runtime.context()notebookutils.runtime.context()
notebookutils.runtime is a utility to manage runtime context. context() returns the runtime context as a list.
notebookutils.runtime.help(methodName = "")notebookutils.runtime.help(methodName = "")
methodName |
method name to get more information.am |
Set runtime high concurrency mode repl id
notebookutils.runtime.setHcReplId(replId)notebookutils.runtime.setHcReplId(replId)
replId |
High concurrency mode repl id |
Stop an interactive session
notebookutils.session.stop(detach = TRUE)notebookutils.session.stop(detach = TRUE)
detach |
If detach is True, stop session from standard session, or detach current notebook from high concurrency session; if detach is False, stop session in any session. Default is TRUE. |
Create a warehouse
notebookutils.warehouse.create( name, description = "", definition = "", workspaceId = "" )notebookutils.warehouse.create( name, description = "", definition = "", workspaceId = "" )
name |
Name of the warehouse |
description |
Description of the warehouse |
definition |
Definition of the warehouse |
workspaceId |
Workspace id of the warehouse, default to current workspace |
A warehouse object
Delete a warehouse
notebookutils.warehouse.delete(name, workspaceId = "")notebookutils.warehouse.delete(name, workspaceId = "")
name |
Name of the warehouse |
workspaceId |
Workspace id of the warehouse, default to current workspace |
Get a warehouse
notebookutils.warehouse.get(name, workspaceId = "")notebookutils.warehouse.get(name, workspaceId = "")
name |
Name of the warehouse |
workspaceId |
Workspace id of the warehouse, default to current workspace |
A warehouse object
Get the definition of a warehouse
notebookutils.warehouse.getDefinition(name, workspaceId = "")notebookutils.warehouse.getDefinition(name, workspaceId = "")
name |
Name of the warehouse |
workspaceId |
Workspace id of the warehouse, default to current workspace |
The definition of the warehouse
notebookutils.warehouse.create(name: String, description: String, workspaceId: String): warehouse -> Create a warehouse notebookutils.warehouse.get(name: String, workspaceId: String): warehouse -> Get a warehouse notebookutils.warehouse.delete(name: String, workspaceId: String): void -> Delete a warehouse notebookutils.warehouse.update(name: String, newName: String, description: String, workspaceId: String): warehouse -> Update a warehouse
notebookutils.warehouse.help(methodName = "")notebookutils.warehouse.help(methodName = "")
methodName |
method name to get more information |
List all warehouses
notebookutils.warehouse.list(workspaceId = "", maxResults = 1000L)notebookutils.warehouse.list(workspaceId = "", maxResults = 1000L)
workspaceId |
Workspace id of the warehouse, default to current workspace |
maxResults |
Maximum number of warehouses to return, default to 1000 |
A list of warehouse objects
Update a warehouse
notebookutils.warehouse.update( name, newName, description = "", workspaceId = "" )notebookutils.warehouse.update( name, newName, description = "", workspaceId = "" )
name |
Name of the warehouse |
newName |
New name of the warehouse |
description |
Description of the warehouse |
workspaceId |
Workspace id of the warehouse, default to current workspace |
A warehouse object
Get the definition of a warehouse
notebookutils.warehouse.updateDefinition(name, definition, workspaceId = "")notebookutils.warehouse.updateDefinition(name, definition, workspaceId = "")
name |
Name of the warehouse |
definition |
Definition of the warehouse |
workspaceId |
Workspace id of the warehouse, default to current workspace |
The definition of the warehouse
Assign a workspace to a capacity
notebookutils.workspace.assignToCapacity(capacityId, workspaceId = "")notebookutils.workspace.assignToCapacity(capacityId, workspaceId = "")
capacityId |
Id of the capacity |
workspaceId |
Id of the workspace, default to current workspace |
Boolean indicating success
Create a workspace
notebookutils.workspace.create(name, description = "", capacityId = "")notebookutils.workspace.create(name, description = "", capacityId = "")
name |
Name of the workspace |
description |
Description of the workspace |
capacityId |
Id of the capacity, default to current capacity |
A workspace object
Delete a workspace
notebookutils.workspace.delete(workspaceId)notebookutils.workspace.delete(workspaceId)
workspaceId |
Id of the workspace |
Get a workspace
notebookutils.workspace.get(name = "")notebookutils.workspace.get(name = "")
name |
Name of the workspace |
A workspace object
notebookutils.workspace.assignToCapacity(capacityId: String, workspaceId: String): Boolean -> Assign a workspace to a capacity notebookutils.workspace.create(name: String, description: String, capacityId: String): workspace -> Create a workspace notebookutils.workspace.delete(workspaceId: String): void -> Delete a workspace notebookutils.workspace.get(name: String): workspace -> Get a workspace notebookutils.workspace.unassignFromCapacity(workspaceId: String): void -> Unassign a workspace from a capacity notebookutils.workspace.update(workspaceId: String, newName: String, description: String): workspace -> Update a workspace
notebookutils.workspace.help(methodName = "")notebookutils.workspace.help(methodName = "")
methodName |
method name to get more information |
List all workspaces
notebookutils.workspace.list(maxResults = 1000L)notebookutils.workspace.list(maxResults = 1000L)
maxResults |
Maximum number of workspaces to return, default to 1000 |
A list of workspace objects
List the specified artifacts in the workspace
notebookutils.workspace.listArtifacts( artifactType, workspaceId = "", maxResults = 1000L )notebookutils.workspace.listArtifacts( artifactType, workspaceId = "", maxResults = 1000L )
artifactType |
Type of the artifact |
workspaceId |
Id of the workspace |
maxResults |
Maximum number of artifacts to return, default to 1000 |
A list of artifact objects
Unassign a workspace from a capacity
notebookutils.workspace.unassignFromCapacity(workspaceId)notebookutils.workspace.unassignFromCapacity(workspaceId)
workspaceId |
Id of the workspace |
Update a workspace
notebookutils.workspace.update(workspaceId, newName, description = "")notebookutils.workspace.update(workspaceId, newName, description = "")
workspaceId |
Id of the workspace |
newName |
New name for the workspace |
description |
New description for the workspace |
Updated workspace object