, Eli Lilly and
Company [cph, fnd]| Title: | A Crew Launcher Plugin for AWS Batch |
|---|---|
| Description: | In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The 'crew.aws.batch' package extends the 'mirai'-powered 'crew' package with a worker launcher plugin for AWS Batch. Inspiration also comes from packages 'mirai' by Gao (2023) <https://github.com/shikokuchuo/mirai>, 'future' by Bengtsson (2021) <doi:10.32614/RJ-2021-048>, 'rrq' by FitzJohn and Ashton (2023) <https://github.com/mrc-ide/rrq>, 'clustermq' by Schubert (2019) <doi:10.1093/bioinformatics/btz284>), and 'batchtools' by Lang, Bischl, and Surmann (2017). <doi:10.21105/joss.00135>. |
| Authors: | William Michael Landau [aut, cre]
, Eli Lilly and
Company [cph, fnd] |
| Maintainer: | William Michael Landau <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.0.8 |
| Built: | 2025-03-07 07:10:58 UTC |
| Source: | CRAN |
In computationally demanding analysis projects,
statisticians and data scientists asynchronously
deploy long-running tasks to distributed systems,
ranging from traditional clusters to cloud services.
The crew.aws.batch package extends the
mirai-powered
crew package with worker
launcher plugins for AWS Batch.
Inspiration also comes from packages
mirai,
future,
rrq,
clustermq,
and batchtools.
AWS Batch definition R6 class
See crew_definition_aws_batch().
In order for the AWS Batch crew job definition class to function
properly, your IAM policy needs permission to perform the
RegisterJobDefinition, DeregisterJobDefinition, and
DescribeJobDefinitions AWS Batch API calls.
For more information on AWS policies and permissions, please visit
https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html.
job_queuejob_definitionlog_groupconfigcredentialsendpointregionnew()
AWS Batch job definition constructor.
crew_class_definition_aws_batch$new( job_queue = NULL, job_definition = NULL, log_group = NULL, config = NULL, credentials = NULL, endpoint = NULL, region = NULL )
job_queuejob_definitionlog_groupconfigcredentialsendpointregionAWS Batch job definition object.
validate()
Validate the object.
crew_class_definition_aws_batch$validate()
NULL (invisibly). Throws an error if a field is invalid.
register()
Register a job definition.
crew_class_definition_aws_batch$register( image, platform_capabilities = "EC2", memory_units = "gigabytes", memory = NULL, cpus = NULL, gpus = NULL, seconds_timeout = NULL, scheduling_priority = NULL, tags = NULL, propagate_tags = NULL, parameters = NULL, job_role_arn = NULL, execution_role_arn = NULL )
imageCharacter of length 1, Docker image used for each job. You can supply a path to an image in Docker Hub or the full URI of an image in an Amazon ECR repository.
platform_capabilitiesOptional character of length 1, either
"EC2" to run on EC2 or "FARGATE" to run on Fargate.
memory_unitsCharacter of length 1,
either "gigabytes" or "mebibytes" to set the units of the
memory argument. "gigabytes" is simpler for EC2 jobs, but
Fargate has strict requirements about specifying exact amounts of
mebibytes (MiB). for details, read
https://docs.aws.amazon.com/cli/latest/reference/batch/register-job-definition.html # nolint
memoryPositive numeric of length 1, amount of memory to request for each job.
cpusPositive numeric of length 1, number of virtual CPUs to request for each job.
gpusPositive numeric of length 1, number of GPUs to request for each job.
seconds_timeoutOptional positive numeric of length 1, number of seconds until a job times out.
scheduling_priorityOptional nonnegative integer of length 1
between 0 and 9999, priority of jobs. Jobs with higher-valued
priorities are scheduled first.
The priority only applies if the job queue has a fair share
policy. Set to NULL to omit.
tagsOptional character vector of tags.
propagate_tagsOptional logical of length 1, whether to propagate tags from the job or definition to the ECS task.
parametersOptional character vector of key-value pairs designating parameters for job submission.
job_role_arnCharacter of length 1, Amazon resource name (ARN) of the job role.
execution_role_arnCharacter of length 1, Amazon resource name (ARN) of the execution role.
The register() method registers a simple
job definition using the job definition name and log group originally
supplied to crew_definition_aws_batch().
Job definitions created with $register() are container-based
and use the AWS log driver.
For more complicated
kinds of jobs, we recommend skipping register(): first call
https://www.paws-r-sdk.com/docs/batch_register_job_definition/
to register the job definition, then supply the job definition
name to the job_definition argument of crew_definition_aws_batch().
A one-row tibble with the job definition name, ARN, and
revision number of the registered job definition.
deregister()
Attempt to deregister a revision of the job definition.
crew_class_definition_aws_batch$deregister(revision = NULL)
revisionFinite positive integer of length 1, optional revision
number to deregister. If NULL, then only the highest revision
number of the job definition is deregistered, if it exists.
Attempt to deregister the job definition whose name was
originally supplied to the job_definition argument of
crew_definition_aws_batch().
NULL (invisibly).
describe()
Describe the revisions of the job definition.
crew_class_definition_aws_batch$describe(revision = NULL, active = FALSE)
revisionPositive integer of length 1, optional revision number to describe.
activeLogical of length 1, whether to filter on just the active job definition.
A tibble with job definition information.
There is one row per revision.
Some fields may be nested lists.
submit()
Submit an AWS Batch job with the given job definition.
crew_class_definition_aws_batch$submit(
command = c("sleep", "300"),
name = paste0("crew-aws-batch-job-", crew::crew_random_name()),
cpus = NULL,
gpus = NULL,
memory_units = "gigabytes",
memory = NULL,
seconds_timeout = NULL,
share_identifier = NULL,
scheduling_priority_override = NULL,
tags = NULL,
propagate_tags = NULL,
parameters = NULL
)commandCharacter vector with the command to submit for the job. Usually a Linux shell command with each term in its own character string.
nameCharacter of length 1 with the job name.
cpusPositive numeric of length 1, number of virtual CPUs to request for each job.
gpusPositive numeric of length 1, number of GPUs to request for each job.
memory_unitsCharacter of length 1,
either "gigabytes" or "mebibytes" to set the units of the
memory argument. "gigabytes" is simpler for EC2 jobs, but
Fargate has strict requirements about specifying exact amounts of
mebibytes (MiB). for details, read
https://docs.aws.amazon.com/cli/latest/reference/batch/register-job-definition.html # nolint
memoryPositive numeric of length 1, amount of memory to request for each job.
seconds_timeoutOptional positive numeric of length 1, number of seconds until a job times out.
share_identifierCharacter of length 1 with the share identifier of the job. Only applies if the job queue has a scheduling policy. Read the official AWS Batch documentation for details.
scheduling_priority_overrideOptional nonnegative integer
of length between 0 and 9999, priority of the job.
This value overrides the priority in the job definition.
Jobs with higher-valued priorities are scheduled first.
The priority applies if the job queue has a fair share policy.
Set to NULL to omit.
tagsOptional character vector of tags.
propagate_tagsOptional logical of length 1, whether to propagate tags from the job or definition to the ECS task.
parametersOptional character vector of key-value pairs designating parameters for job submission.
This method uses the job queue and job definition
that were supplied through crew_definition_aws_batch().
Any jobs submitted this way are different from the
crew workers that the crew controller starts automatically
using the AWS Batch launcher plugin.
You may use the submit() method in the definition for different
purposes such as testing.
A one-row tibble with the name, ID, and
Amazon resource name (ARN) of the job.
Other definition:
crew_definition_aws_batch()
AWS Batch launcher R6 class
See crew_launcher_aws_batch().
In order for the AWS Batch crew plugin to function properly, your IAM
policy needs permission to perform the SubmitJob and TerminateJob
AWS Batch API calls. For more information on AWS policies and permissions,
please visit
https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html.
The AWS Batch controller and launcher accept many arguments
which start with "aws_batch_". These arguments are AWS-Batch-specific
parameters forwarded directly to the submit_job() method for
the Batch client in the paws.compute R package
For a full description of each argument, including its meaning and format, please visit https://www.paws-r-sdk.com/docs/batch_submit_job/. The upstream API documentation is at https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html and the analogous CLI documentation is at https://docs.aws.amazon.com/cli/latest/reference/batch/submit-job.html.
The actual argument names may vary slightly, depending
on which : for example, the aws_batch_job_definition argument of
the crew AWS Batch launcher/controller corresponds to the
jobDefinition argument of the web API and
paws.compute::batch()$submit_job(), and both correspond to the
--job-definition argument of the CLI.
Control verbosity with the paws.log_level global option in R.
Set to 0 for minimum verbosity and 3 for maximum verbosity.
crew::crew_class_launcher -> crew_class_launcher_aws_batch
options_aws_batchcrew::crew_class_launcher$call()crew::crew_class_launcher$crashes()crew::crew_class_launcher$launch()crew::crew_class_launcher$poll()crew::crew_class_launcher$resolve()crew::crew_class_launcher$scale()crew::crew_class_launcher$set_name()crew::crew_class_launcher$settings()crew::crew_class_launcher$start()crew::crew_class_launcher$terminate()crew::crew_class_launcher$terminate_workers()crew::crew_class_launcher$update()new()
Abstract launcher constructor.
crew_class_launcher_aws_batch$new( name = NULL, workers = NULL, seconds_interval = NULL, seconds_timeout = NULL, seconds_launch = NULL, seconds_idle = NULL, seconds_wall = NULL, tasks_max = NULL, tasks_timers = NULL, reset_globals = NULL, reset_packages = NULL, reset_options = NULL, garbage_collection = NULL, tls = NULL, processes = NULL, r_arguments = NULL, options_metrics = NULL, options_aws_batch = NULL )
nameworkersseconds_intervalseconds_timeoutseconds_launchseconds_idleseconds_walltasks_maxtasks_timersreset_globalsreset_packagesreset_optionsgarbage_collectiontlsprocessesr_argumentsoptions_metricsoptions_aws_batchAn abstract launcher object.
validate()
Validate the launcher.
crew_class_launcher_aws_batch$validate()
NULL (invisibly). Throws an error if a field is invalid.
launch_worker()
Launch a local process worker which will dial into a socket.
crew_class_launcher_aws_batch$launch_worker(call, name, launcher, worker)
callCharacter string, a namespaced call to
crew::crew_worker()
which will run in the worker and accept tasks.
nameCharacter string, an informative worker name.
launcherCharacter string, name of the launcher.
workerCharacter string, name of the worker instance.
The call argument is R code that will run to
initiate the worker.
A handle object to allow the termination of the worker later on.
terminate_worker()
Terminate a local process worker.
crew_class_launcher_aws_batch$terminate_worker(handle)
handleA process handle object previously
returned by launch_worker().
NULL (invisibly).
Other plugin_aws_batch:
crew_controller_aws_batch(),
crew_launcher_aws_batch()
AWS Batch monitor R6 class
In order for the AWS Batch crew monitor class to function
properly, your IAM policy needs permission to perform the SubmitJob,
TerminateJob, ListJobs, and DescribeJobs AWS Batch API calls.
In addition, to download CloudWatch logs with the log() method,
your IAM policy also needs permission to perform the GetLogEvents
CloudWatch logs API call.
For more information on AWS policies and permissions, please visit
https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html.
job_queuejob_definitionlog_groupconfigcredentialsendpointregionnew()
AWS Batch job definition constructor.
crew_class_monitor_aws_batch$new( job_queue = NULL, job_definition = NULL, log_group = NULL, config = NULL, credentials = NULL, endpoint = NULL, region = NULL )
job_queuejob_definitionlog_groupconfigcredentialsendpointregionAWS Batch job definition object.
validate()
Validate the object.
crew_class_monitor_aws_batch$validate()
NULL (invisibly). Throws an error if a field is invalid.
terminate()
Terminate one or more AWS Batch jobs.
crew_class_monitor_aws_batch$terminate( ids = NULL, all = FALSE, reason = "cancelled/terminated by crew.aws.batch monitor", verbose = TRUE )
idsCharacter vector with the IDs of the AWS Batch jobs
to terminate. Leave as NULL if all is TRUE.
allTRUE to terminate all jobs belonging to
the previously specified job definition. FALSE to terminate
only the job IDs given in the ids argument.
reasonCharacter of length 1, natural language explaining the reason the job was terminated.
verboseLogical of length 1, whether to show a progress bar
if the R process is interactive and length(ids) is greater than 1.
NULL (invisibly).
status()
Get the status of a single job
crew_class_monitor_aws_batch$status(id)
idCharacter of length 1, job ID. This is different from the user-supplied job name.
A one-row tibble with information about the job.
log()
Get the CloudWatch log of a job.
crew_class_monitor_aws_batch$log(id, path = stdout(), start_from_head = FALSE)
idCharacter of length 1, job ID. This is different from the user-supplied job name.
pathCharacter string or stream (e.g. stdout()),
file path or connection passed to the con argument of
writeLines() to print the log messages.
Set to nullfile() to suppress output
(and use the invisibly returned tibble object instead).
start_from_headLogical of length 1, whether to print earlier log events before later ones.
This method assumes the job has log driver "awslogs"
(specifying AWS CloudWatch) and that the log group is the one
prespecified in the log_group argument of
crew_monitor_aws_batch(). This method cannot use
other log drivers such as Splunk, and it will fail if the log
group is wrong or missing.
log() invisibly returns a tibble with log information
and writes the messages to the stream or path given by the
path argument.
jobs()
List all the jobs in the given job queue with the given job definition.
crew_class_monitor_aws_batch$jobs(
status = c("submitted", "pending", "runnable", "starting", "running", "succeeded",
"failed")
)statusCharacter vector of job states. Results are limited to these job states.
The output only includes jobs under the
job queue and job definition
that were supplied through crew_monitor_aws_batch().
A tibble with one row per job and columns
with job information.
active()
List active jobs: submitted, pending, runnable, starting, or running (not succeeded or failed).
crew_class_monitor_aws_batch$active()
The output only includes jobs under the
job queue and job definition
that were supplied through crew_monitor_aws_batch().
A tibble with one row per job and columns
with job information.
inactive()
List inactive jobs: ones whose status is succeeded or failed (not submitted, pending, runnable, starting, or running).
crew_class_monitor_aws_batch$inactive()
The output only includes jobs under the
job queue and job definition
that were supplied through crew_monitor_aws_batch().
A tibble with one row per job and columns
with job information.
submitted()
List jobs whose status is "submitted".
crew_class_monitor_aws_batch$submitted()
The output only includes jobs under the
job queue and job definition
that were supplied through crew_monitor_aws_batch().
A tibble with one row per job and columns
with job information.
pending()
List jobs whose status is "pending".
crew_class_monitor_aws_batch$pending()
The output only includes jobs under the
job queue and job definition
that were supplied through crew_monitor_aws_batch().
A tibble with one row per job and columns
with job information.
runnable()
List jobs whose status is "runnable".
crew_class_monitor_aws_batch$runnable()
The output only includes jobs under the
job queue and job definition
that were supplied through crew_monitor_aws_batch().
A tibble with one row per job and columns
with job information.
starting()
List jobs whose status is "starting".
crew_class_monitor_aws_batch$starting()
The output only includes jobs under the
job queue and job definition
that were supplied through crew_monitor_aws_batch().
A tibble with one row per job and columns
with job information.
running()
List jobs whose status is "running".
crew_class_monitor_aws_batch$running()
The output only includes jobs under the
job queue and job definition
that were supplied through crew_monitor_aws_batch().
A tibble with one row per job and columns
with job information.
succeeded()
List jobs whose status is "succeeded".
crew_class_monitor_aws_batch$succeeded()
The output only includes jobs under the
job queue and job definition
that were supplied through crew_monitor_aws_batch().
A tibble with one row per job and columns
with job information.
failed()
List jobs whose status is "failed".
crew_class_monitor_aws_batch$failed()
The output only includes jobs under the
job queue and job definition
that were supplied through crew_monitor_aws_batch().
A tibble with one row per job and columns
with job information.
Other monitor:
crew_monitor_aws_batch()
Create an R6 object to submit tasks and
launch workers on AWS Batch workers.
crew_controller_aws_batch( name = NULL, workers = 1L, host = NULL, port = NULL, tls = crew::crew_tls(mode = "automatic"), tls_enable = NULL, tls_config = NULL, seconds_interval = 0.5, seconds_timeout = 60, seconds_launch = 1800, seconds_idle = 300, seconds_wall = Inf, retry_tasks = NULL, tasks_max = Inf, tasks_timers = 0L, reset_globals = TRUE, reset_packages = FALSE, reset_options = FALSE, garbage_collection = FALSE, crashes_error = NULL, processes = NULL, r_arguments = c("--no-save", "--no-restore"), crashes_max = 5L, backup = NULL, options_metrics = crew::crew_options_metrics(), options_aws_batch = crew.aws.batch::crew_options_aws_batch(), aws_batch_config = NULL, aws_batch_credentials = NULL, aws_batch_endpoint = NULL, aws_batch_region = NULL, aws_batch_job_definition = NULL, aws_batch_job_queue = NULL, aws_batch_share_identifier = NULL, aws_batch_scheduling_priority_override = NULL, aws_batch_parameters = NULL, aws_batch_container_overrides = NULL, aws_batch_node_overrides = NULL, aws_batch_retry_strategy = NULL, aws_batch_propagate_tags = NULL, aws_batch_timeout = NULL, aws_batch_tags = NULL, aws_batch_eks_properties_override = NULL )crew_controller_aws_batch( name = NULL, workers = 1L, host = NULL, port = NULL, tls = crew::crew_tls(mode = "automatic"), tls_enable = NULL, tls_config = NULL, seconds_interval = 0.5, seconds_timeout = 60, seconds_launch = 1800, seconds_idle = 300, seconds_wall = Inf, retry_tasks = NULL, tasks_max = Inf, tasks_timers = 0L, reset_globals = TRUE, reset_packages = FALSE, reset_options = FALSE, garbage_collection = FALSE, crashes_error = NULL, processes = NULL, r_arguments = c("--no-save", "--no-restore"), crashes_max = 5L, backup = NULL, options_metrics = crew::crew_options_metrics(), options_aws_batch = crew.aws.batch::crew_options_aws_batch(), aws_batch_config = NULL, aws_batch_credentials = NULL, aws_batch_endpoint = NULL, aws_batch_region = NULL, aws_batch_job_definition = NULL, aws_batch_job_queue = NULL, aws_batch_share_identifier = NULL, aws_batch_scheduling_priority_override = NULL, aws_batch_parameters = NULL, aws_batch_container_overrides = NULL, aws_batch_node_overrides = NULL, aws_batch_retry_strategy = NULL, aws_batch_propagate_tags = NULL, aws_batch_timeout = NULL, aws_batch_tags = NULL, aws_batch_eks_properties_override = NULL )
name |
Deprecated on 2025-01-14 ( |
workers |
Deprecated on 2025-01-13 ( |
host |
IP address of the |
port |
TCP port to listen for the workers. If |
tls |
A TLS configuration object from |
tls_enable |
Deprecated on 2023-09-15 in version 0.4.1.
Use argument |
tls_config |
Deprecated on 2023-09-15 in version 0.4.1.
Use argument |
seconds_interval |
Number of seconds between
polling intervals waiting for certain internal
synchronous operations to complete,
such as checking |
seconds_timeout |
Number of seconds until timing
out while waiting for certain synchronous operations to complete,
such as checking |
seconds_launch |
Seconds of startup time to allow.
A worker is unconditionally assumed to be alive
from the moment of its launch until |
seconds_idle |
Maximum number of seconds that a worker can idle
since the completion of the last task. If exceeded, the worker exits.
But the timer does not launch until |
seconds_wall |
Soft wall time in seconds.
The timer does not launch until |
retry_tasks |
Deprecated on 2025-01-13 ( |
tasks_max |
Maximum number of tasks that a worker will do before
exiting. See the |
tasks_timers |
Number of tasks to do before activating
the timers for |
reset_globals |
|
reset_packages |
|
reset_options |
|
garbage_collection |
|
crashes_error |
Deprecated on 2025-01-13 ( |
processes |
|
r_arguments |
Optional character vector of command line arguments
to pass to |
crashes_max |
In rare cases, a worker may exit unexpectedly
before it completes its current task. If this happens,
|
backup |
An optional Limitations of |
options_metrics |
Either |
options_aws_batch |
List of options from |
aws_batch_config |
Deprecated.
Use |
aws_batch_credentials |
Deprecated.
Use |
aws_batch_endpoint |
Deprecated.
Use |
aws_batch_region |
Deprecated.
Use |
aws_batch_job_definition |
Deprecated.
Use |
aws_batch_job_queue |
Deprecated.
Use |
aws_batch_share_identifier |
Deprecated.
Use |
aws_batch_scheduling_priority_override |
Deprecated.
Use |
aws_batch_parameters |
Deprecated.
Use |
aws_batch_container_overrides |
Deprecated.
Use |
aws_batch_node_overrides |
Deprecated.
Use |
aws_batch_retry_strategy |
Deprecated.
Use |
aws_batch_propagate_tags |
Deprecated.
Use |
aws_batch_timeout |
Deprecated.
Use |
aws_batch_tags |
Deprecated.
Use |
aws_batch_eks_properties_override |
Deprecated.
Use |
In order for the AWS Batch crew plugin to function properly, your IAM
policy needs permission to perform the SubmitJob and TerminateJob
AWS Batch API calls. For more information on AWS policies and permissions,
please visit
https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html.
The AWS Batch controller and launcher accept many arguments
which start with "aws_batch_". These arguments are AWS-Batch-specific
parameters forwarded directly to the submit_job() method for
the Batch client in the paws.compute R package
For a full description of each argument, including its meaning and format, please visit https://www.paws-r-sdk.com/docs/batch_submit_job/. The upstream API documentation is at https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html and the analogous CLI documentation is at https://docs.aws.amazon.com/cli/latest/reference/batch/submit-job.html.
The actual argument names may vary slightly, depending
on which : for example, the aws_batch_job_definition argument of
the crew AWS Batch launcher/controller corresponds to the
jobDefinition argument of the web API and
paws.compute::batch()$submit_job(), and both correspond to the
--job-definition argument of the CLI.
Control verbosity with the paws.log_level global option in R.
Set to 0 for minimum verbosity and 3 for maximum verbosity.
Other plugin_aws_batch:
crew_class_launcher_aws_batch,
crew_launcher_aws_batch()
if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) { controller <- crew_controller_aws_batch( aws_batch_job_definition = "YOUR_JOB_DEFINITION_NAME", aws_batch_job_queue = "YOUR_JOB_QUEUE_NAME" ) controller$start() controller$push(name = "task", command = sqrt(4)) controller$wait() controller$pop()$result controller$terminate() }if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) { controller <- crew_controller_aws_batch( aws_batch_job_definition = "YOUR_JOB_DEFINITION_NAME", aws_batch_job_queue = "YOUR_JOB_QUEUE_NAME" ) controller$start() controller$push(name = "task", command = sqrt(4)) controller$wait() controller$pop()$result controller$terminate() }
Create an R6 object to manage a job definition for AWS
Batch jobs.
crew_definition_aws_batch( job_queue, job_definition = paste0("crew-aws-batch-job-definition-", crew::crew_random_name()), log_group = "/aws/batch/job", config = NULL, credentials = NULL, endpoint = NULL, region = NULL )crew_definition_aws_batch( job_queue, job_definition = paste0("crew-aws-batch-job-definition-", crew::crew_random_name()), log_group = "/aws/batch/job", config = NULL, credentials = NULL, endpoint = NULL, region = NULL )
job_queue |
Character vector of names of AWS Batch job queues.
As of |
job_definition |
Character of length 1, name of the AWS Batch
job definition. The job definition might or might not exist
at the time |
log_group |
Character of length 1,
AWS Batch CloudWatch log group to get job logs.
The default log group is often |
config |
Optional named list, |
credentials |
Optional named list. |
endpoint |
Optional character of length 1. |
region |
Character of length 1. |
An R6 job definition object.
In order for the AWS Batch crew job definition class to function
properly, your IAM policy needs permission to perform the
RegisterJobDefinition, DeregisterJobDefinition, and
DescribeJobDefinitions AWS Batch API calls.
For more information on AWS policies and permissions, please visit
https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html.
Other definition:
crew_class_definition_aws_batch
Create an R6 AWS Batch launcher object.
crew_launcher_aws_batch( name = NULL, workers = 1L, seconds_interval = 0.5, seconds_timeout = 60, seconds_launch = 1800, seconds_idle = 300, seconds_wall = Inf, tasks_max = Inf, tasks_timers = 0L, reset_globals = TRUE, reset_packages = FALSE, reset_options = FALSE, garbage_collection = FALSE, crashes_error = NULL, tls = crew::crew_tls(mode = "automatic"), processes = NULL, r_arguments = c("--no-save", "--no-restore"), options_metrics = crew::crew_options_metrics(), options_aws_batch = crew.aws.batch::crew_options_aws_batch(), aws_batch_config = NULL, aws_batch_credentials = NULL, aws_batch_endpoint = NULL, aws_batch_region = NULL, aws_batch_job_definition = NULL, aws_batch_job_queue = NULL, aws_batch_share_identifier = NULL, aws_batch_scheduling_priority_override = NULL, aws_batch_parameters = NULL, aws_batch_container_overrides = NULL, aws_batch_node_overrides = NULL, aws_batch_retry_strategy = NULL, aws_batch_propagate_tags = NULL, aws_batch_timeout = NULL, aws_batch_tags = NULL, aws_batch_eks_properties_override = NULL )crew_launcher_aws_batch( name = NULL, workers = 1L, seconds_interval = 0.5, seconds_timeout = 60, seconds_launch = 1800, seconds_idle = 300, seconds_wall = Inf, tasks_max = Inf, tasks_timers = 0L, reset_globals = TRUE, reset_packages = FALSE, reset_options = FALSE, garbage_collection = FALSE, crashes_error = NULL, tls = crew::crew_tls(mode = "automatic"), processes = NULL, r_arguments = c("--no-save", "--no-restore"), options_metrics = crew::crew_options_metrics(), options_aws_batch = crew.aws.batch::crew_options_aws_batch(), aws_batch_config = NULL, aws_batch_credentials = NULL, aws_batch_endpoint = NULL, aws_batch_region = NULL, aws_batch_job_definition = NULL, aws_batch_job_queue = NULL, aws_batch_share_identifier = NULL, aws_batch_scheduling_priority_override = NULL, aws_batch_parameters = NULL, aws_batch_container_overrides = NULL, aws_batch_node_overrides = NULL, aws_batch_retry_strategy = NULL, aws_batch_propagate_tags = NULL, aws_batch_timeout = NULL, aws_batch_tags = NULL, aws_batch_eks_properties_override = NULL )
name |
Character string, name of the launcher. If the name is
|
workers |
Maximum number of workers to run concurrently
when auto-scaling, excluding task retries and manual calls to |
seconds_interval |
Number of seconds between
polling intervals waiting for certain internal
synchronous operations to complete. In certain cases, exponential
backoff is used with this argument passed to |
seconds_timeout |
Number of seconds until timing
out while waiting for certain synchronous operations to complete,
such as checking |
seconds_launch |
Seconds of startup time to allow.
A worker is unconditionally assumed to be alive
from the moment of its launch until |
seconds_idle |
Maximum number of seconds that a worker can idle
since the completion of the last task. If exceeded, the worker exits.
But the timer does not launch until |
seconds_wall |
Soft wall time in seconds.
The timer does not launch until |
tasks_max |
Maximum number of tasks that a worker will do before
exiting. See the |
tasks_timers |
Number of tasks to do before activating
the timers for |
reset_globals |
|
reset_packages |
|
reset_options |
|
garbage_collection |
|
crashes_error |
Deprecated on 2025-01-13 ( |
tls |
A TLS configuration object from |
processes |
|
r_arguments |
Optional character vector of command line arguments
to pass to |
options_metrics |
Either |
options_aws_batch |
List of options from |
aws_batch_config |
Deprecated.
Use |
aws_batch_credentials |
Deprecated.
Use |
aws_batch_endpoint |
Deprecated.
Use |
aws_batch_region |
Deprecated.
Use |
aws_batch_job_definition |
Deprecated.
Use |
aws_batch_job_queue |
Deprecated.
Use |
aws_batch_share_identifier |
Deprecated.
Use |
aws_batch_scheduling_priority_override |
Deprecated.
Use |
aws_batch_parameters |
Deprecated.
Use |
aws_batch_container_overrides |
Deprecated.
Use |
aws_batch_node_overrides |
Deprecated.
Use |
aws_batch_retry_strategy |
Deprecated.
Use |
aws_batch_propagate_tags |
Deprecated.
Use |
aws_batch_timeout |
Deprecated.
Use |
aws_batch_tags |
Deprecated.
Use |
aws_batch_eks_properties_override |
Deprecated.
Use |
An R6 AWS Batch launcher object.
In order for the AWS Batch crew plugin to function properly, your IAM
policy needs permission to perform the SubmitJob and TerminateJob
AWS Batch API calls. For more information on AWS policies and permissions,
please visit
https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html.
The AWS Batch controller and launcher accept many arguments
which start with "aws_batch_". These arguments are AWS-Batch-specific
parameters forwarded directly to the submit_job() method for
the Batch client in the paws.compute R package
For a full description of each argument, including its meaning and format, please visit https://www.paws-r-sdk.com/docs/batch_submit_job/. The upstream API documentation is at https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html and the analogous CLI documentation is at https://docs.aws.amazon.com/cli/latest/reference/batch/submit-job.html.
The actual argument names may vary slightly, depending
on which : for example, the aws_batch_job_definition argument of
the crew AWS Batch launcher/controller corresponds to the
jobDefinition argument of the web API and
paws.compute::batch()$submit_job(), and both correspond to the
--job-definition argument of the CLI.
Control verbosity with the paws.log_level global option in R.
Set to 0 for minimum verbosity and 3 for maximum verbosity.
Other plugin_aws_batch:
crew_class_launcher_aws_batch,
crew_controller_aws_batch()
Create an R6 object to list, inspect, and terminate
AWS Batch jobs.
crew_monitor_aws_batch( job_queue, job_definition, log_group = "/aws/batch/job", config = NULL, credentials = NULL, endpoint = NULL, region = NULL )crew_monitor_aws_batch( job_queue, job_definition, log_group = "/aws/batch/job", config = NULL, credentials = NULL, endpoint = NULL, region = NULL )
job_queue |
Character vector of names of AWS Batch job queues.
As of |
job_definition |
Character string, name of the AWS Batch job definition. |
log_group |
Character of length 1,
AWS Batch CloudWatch log group to get job logs.
The default log group is often |
config |
Optional named list, |
credentials |
Optional named list. |
endpoint |
Optional character of length 1. |
region |
Character of length 1. |
In order for the AWS Batch crew monitor class to function
properly, your IAM policy needs permission to perform the SubmitJob,
TerminateJob, ListJobs, and DescribeJobs AWS Batch API calls.
In addition, to download CloudWatch logs with the log() method,
your IAM policy also needs permission to perform the GetLogEvents
CloudWatch logs API call.
For more information on AWS policies and permissions, please visit
https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html.
Other monitor:
crew_class_monitor_aws_batch
Options for the AWS Batch controller.
crew_options_aws_batch( job_definition = "example", job_queue = "example", cpus = NULL, gpus = NULL, memory = NULL, memory_units = "gigabytes", config = list(), credentials = list(), endpoint = NULL, region = NULL, share_identifier = NULL, scheduling_priority_override = NULL, parameters = NULL, container_overrides = NULL, node_overrides = NULL, retry_strategy = NULL, propagate_tags = NULL, timeout = NULL, tags = NULL, eks_properties_override = NULL, verbose = FALSE )crew_options_aws_batch( job_definition = "example", job_queue = "example", cpus = NULL, gpus = NULL, memory = NULL, memory_units = "gigabytes", config = list(), credentials = list(), endpoint = NULL, region = NULL, share_identifier = NULL, scheduling_priority_override = NULL, parameters = NULL, container_overrides = NULL, node_overrides = NULL, retry_strategy = NULL, propagate_tags = NULL, timeout = NULL, tags = NULL, eks_properties_override = NULL, verbose = FALSE )
job_definition |
Character of length 1, name of the AWS Batch job definition to use. There is no default for this argument, and a job definition must be created prior to running the controller. Please see https://docs.aws.amazon.com/batch/ for details. To create a job definition, you will need to create a Docker-compatible
image which can run R and |
job_queue |
Character of length 1, name of the AWS Batch job queue to use. There is no default for this argument, and a job queue must be created prior to running the controller. Please see https://docs.aws.amazon.com/batch/ for details. |
cpus |
Positive numeric scalar,
number of virtual CPUs to request per job. Can be |
gpus |
Positive numeric scalar,
number of GPUs to request per job. Can be |
memory |
Positive numeric scalar,
amount of random access memory (RAM) to request per job.
Choose the units of memory with the |
memory_units |
Character string, units of memory of the |
config |
Named list, |
credentials |
Named list. |
endpoint |
Character of length 1. |
region |
Character of length 1. |
share_identifier |
|
scheduling_priority_override |
|
parameters |
|
container_overrides |
|
node_overrides |
|
retry_strategy |
|
propagate_tags |
|
timeout |
|
tags |
|
eks_properties_override |
|
verbose |
|
A classed list of options for the controller.
Retryable options are deprecated in crew.aws.batch
as of 2025-01-27 (version 0.0.8).