Package: shard 0.1.1

Bradley Buchsbaum

shard: Deterministic, Zero-Copy Parallel Execution for R

Provides a parallel execution runtime for R that emphasizes deterministic memory behavior and efficient handling of large shared inputs. 'shard' enables zero-copy parallel reads via shared/memory-mapped segments, encourages explicit output buffers to avoid large result aggregation, and supervises worker processes to mitigate memory drift via controlled recycling. Diagnostics report peak memory usage, end-of-run memory return, and hidden copy/materialization events to support reproducible performance benchmarking.

Authors:Bradley Buchsbaum [aut, cre, cph]

shard_0.1.1.tar.gz
shard_0.1.1.tar.gz(r-4.7-arm64)shard_0.1.1.tar.gz(r-4.7-x86_64)shard_0.1.1.tar.gz(r-4.6-arm64)shard_0.1.1.tar.gz(r-4.6-x86_64)
shard_0.1.1.tgz(r-4.6-emscripten)
manual.pdf |manual.html
card.svg |card.png
shard/json (API)

# Install 'shard' in R:
install.packages('shard', repos = c('https://cran.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/bbuchsbaum/shard/issues

Pkgdown/docs site:https://bbuchsbaum.github.io

Uses libs:
  • openblas– Optimized BLAS

On CRAN:

Conda:

openblas

2.70 score 3 scripts 491 downloads 107 exports 0 dependencies

Last updated from:5be9b725f7. Checks:6 OK. Indexed: no.

TargetResultTimeFilesSyslog
linux-devel-arm64OK305
linux-devel-x86_64OK285
source / vignettesOK222
linux-release-arm64OK304
linux-release-x86_64OK277
wasm-releaseOK96

Exports:affinity_supportedarenaarena_depthas_sharedas_tibbleavailable_backingsboolbufferbuffer_advisebuffer_closebuffer_diagnosticsbuffer_infobuffer_openbuffer_pathcollectcopy_reportcow_reportdispatch_chunksfactor_colfetchfloat64idx_rangein_arenaint32is_block_viewis_sharedis_shared_vectoris_viewis_windowsiterate_row_groupslist_kernelsmaterializemem_reportpin_workerspool_createpool_dispatchpool_getpool_health_checkpool_lapplypool_sapplypool_statuspool_stopraw_colrecommendationsregister_kernelreportresultsrow_layoutschemascratch_diagnosticsscratch_matrixscratch_pool_configsegment_advisesegment_closesegment_createsegment_infosegment_opensegment_pathsegment_protectsegment_readsegment_reportsegment_sizesegment_writeset_affinityshard_apply_matrixshard_apply_policyshard_crossprodshard_get_adaptershard_lapply_sharedshard_list_adaptersshard_mapshard_reduceshard_register_adaptershard_share_hookshard_unregister_adaptershardsshards_listshareshare_openshared_adviseshared_diagnosticsshared_infoshared_reset_diagnosticsshared_segmentshared_vectorshared_viewstream_countstream_filterstream_group_countstream_group_sumstream_mapstream_reducestream_sumstream_top_kstring_colsucceededtable_buffertable_diagnosticstable_finalizetable_sinktable_writetask_reportviewview_blockview_diagnosticsview_gatherview_info

Dependencies:

Getting Started with shard

Rendered fromshard.Rmdusingknitr::rmarkdownon Jun 04 2026.

Last update: 2026-04-03
Started: 2026-04-03

Readme and manuals

Help Manual

Help pageTopics
Extract Buffer Elements[.shard_buffer
Subset Shard Descriptor[.shard_descriptor
Subset a shard_descriptor_lazy Object[.shard_descriptor_lazy
Get Single Shard[[.shard_descriptor
Extract a Single Shard from a shard_descriptor_lazy Object[[.shard_descriptor_lazy
Double-bracket Subset-assign a Shared Vector[[<-.shard_shared_vector
Assign to Buffer Elements[<-.shard_buffer
Subset-assign a Shared Vector[<-.shard_shared_vector
Adapter Registry for Class-Specific Deep Sharingadapter
CPU Affinity + mmap Advice (Advanced)affinity
Check whether CPU affinity is supportedaffinity_supported
ALTREP Shared Vectorsaltrep
Arena Semantic Scopearena
Get Current Arena Deptharena_depth
Create a shared vector from an existing R vectoras_shared
Materialize a shard table handle as a data.frame/tibbleas_tibble
Materialize a dataset handle into a data.frame/tibbleas_tibble.shard_dataset
Materialize a row-groups handle into a data.frame/tibbleas_tibble.shard_row_groups
Materialize a fixed table handle or bufferas_tibble.shard_table_buffer
Materialize a table handle into a data.frame/tibbleas_tibble.shard_table_handle
Coerce a Shared Memory Buffer to Arrayas.array.shard_buffer
Coerce a Shared Memory Buffer to Doubleas.double.shard_buffer
Coerce a Shared Memory Buffer to Integeras.integer.shard_buffer
Coerce a Shared Memory Buffer to Logicalas.logical.shard_buffer
Coerce a Shared Memory Buffer to Matrixas.matrix.shard_buffer
Coerce a Shared Memory Buffer to Rawas.raw.shard_buffer
Coerce a Shared Memory Buffer to a Vectoras.vector.shard_buffer
Set an Attribute on a Shared Vectorattr<-.shard_shared_vector
Set Attributes on a Shared Vectorattributes<-.shard_shared_vector
Get available shared memory backing typesavailable_backings
Shared Memory Buffersbuffer
Advise access pattern for a bufferbuffer_advise
Close a Bufferbuffer_close
Buffer Diagnosticsbuffer_diagnostics
Get Buffer Infobuffer_info
Open an Existing Bufferbuffer_open
Get Buffer Pathbuffer_path
Close a Shared Objectclose.shard_deep_shared close.shard_shared close.shard_shared_vector
Collect a shard table into memorycollect
Collect a dataset handle into memorycollect.shard_dataset
Collect a row-groups handle into memorycollect.shard_row_groups
Collect a table handle into memorycollect.shard_table_handle
Column Typesbool coltypes float64 int32 raw_col string_col
Data Copy Reportcopy_report
Copy-on-Write Policy Reportcow_report
Diagnostics APIdiagnostics
Dimensions of a Shared Memory Bufferdim.shard_buffer
Set dim on a Shared Vectordim<-.shard_shared_vector
Set dimnames on a Shared Vectordimnames<-.shard_shared_vector
Task Dispatch Enginedispatch
Dispatch Chunks to Worker Pooldispatch_chunks
Ergonomic Apply/Lapply Wrappersergonomics
Categorical column typefactor_col
Fetch Data from a Shared Objectfetch fetch.default fetch.shard_deep_shared fetch.shard_shared
Contiguous index rangeidx_range
Check if Currently Inside an Arenain_arena
Check if Object is Sharedis_shared
Check if an object is a shared vectoris_shared_vector
View Predicatesis_block_view is_view
Check if running on Windowsis_windows
Iterate row groupsiterate_row_groups
Length of a Shared Memory Bufferlength.shard_buffer
Length of a shard_descriptor Objectlength.shard_descriptor
Length of a shard_descriptor_lazy Objectlength.shard_descriptor_lazy
List registered kernelslist_kernels
Materialize Shared Objectmaterialize materialize.default materialize.shard_shared
Materialize a block view into an R matrixmaterialize.shard_view_block
Materialize a gather view into an R matrixmaterialize.shard_view_gather
Memory Usage Reportmem_report
Set Names on a Shared Vectornames<-.shard_shared_vector
Pin shard workers to CPU corespin_workers
Worker Pool Managementpool
Create a Worker Poolpool_create
Dispatch Task to Workerpool_dispatch
Get the Current Worker Poolpool_get
Check Pool Healthpool_health_check
Parallel Dispatch with Async Workerspool_lapply
Parallel sapply with Supervisionpool_sapply
Get Pool Statuspool_status
Stop the Worker Poolpool_stop
Print an arena_result objectprint.arena_result
Print a shard_apply_policy Objectprint.shard_apply_policy
Print a Shared Memory Bufferprint.shard_buffer
Print a Deep-Shared Objectprint.shard_deep_shared
Print a shard_descriptor Objectprint.shard_descriptor
Print a shard_descriptor_lazy Objectprint.shard_descriptor_lazy
Print a shard_dispatch_result Objectprint.shard_dispatch_result
Print a shard_health_report Objectprint.shard_health_report
Print a shard_idx_range objectprint.shard_idx_range
Print a shard_pool Objectprint.shard_pool
Print a shard_reduce_result Objectprint.shard_reduce_result
Print a shard_report Objectprint.shard_report
Print a shard_result Objectprint.shard_result
Print a Shared Memory Segmentprint.shard_segment
Print a Shared Objectprint.shard_shared
Print a Shared Vectorprint.shard_shared_vector
Print a shard_tiles objectprint.shard_tiles
Print a shard_view_block objectprint.shard_view_block
Print a shard_view_gather objectprint.shard_view_gather
Print a shard_worker Objectprint.shard_worker
Chunk Queue Managementqueue
Performance Recommendationsrecommendations
Register a shard kernelregister_kernel
Generate Shard Runtime Reportreport
Extract Results from shard_mapresults
Row layout for fixed-row table outputsrow_layout
RSS Monitoring Utilitiesrss
Define a table schemaschema
Scratch pool diagnosticsscratch_diagnostics
Get a scratch matrixscratch_matrix
Configure scratch pool limitsscratch_pool_config
Shared Memory Segmentsegment
Advise OS about expected access pattern for a segmentsegment_advise
Close a shared memory segmentsegment_close
Create a new shared memory segmentsegment_create
Get segment informationsegment_info
Open an existing shared memory segmentsegment_open
Get the path or name of a segmentsegment_path
Make a segment read-onlysegment_protect
Read raw data from a segmentsegment_read
Shared Memory Segment Reportsegment_report
Get the size of a segmentsegment_size
Write data to a segmentsegment_write
Set CPU affinity for the current processset_affinity
Apply a Function Over Matrix Columns with Shared Inputsshard_apply_matrix
Apply Wrapper Policyshard_apply_policy
Parallel crossprod() using shard views + output buffersshard_crossprod
Get Adapter for an Objectshard_get_adapter
Apply a Function Over a List with Optional Auto-Sharingshard_lapply_shared
List Registered Adaptersshard_list_adapters
Parallel Execution with shard_mapshard_map
Streaming Reductions over Shardsshard_reduce
Register an Adapter for Class-Specific Traversalshard_register_adapter
Deep Sharing Hook for Custom Classesshard_share_hook shard_share_hook.default
Unregister an Adaptershard_unregister_adapter
Shard Descriptor Creationshards
Create Shards from an Explicit Index Listshards_list
Zero-Copy Shared Objectsshare
Open an Existing Shared Object by Pathshare_open
Advise access pattern for a shared input vector/matrixshared_advise
Get diagnostics for a shared vectorshared_diagnostics
Get Information About a Shared Objectshared_info
Reset diagnostic counters for a shared vectorshared_reset_diagnostics
Get the underlying segment from a shared vectorshared_segment
Create a shared vector from a segmentshared_vector
Create a view (subset) of a shared vectorshared_view
Stream row countstream_count
Stream-filter a dataset/row-groups into a new partitioned datasetstream_filter
Stream group-wise countstream_group_count
Stream group-wise sumstream_group_sum
Stream over row-groups/datasets and mapstream_map stream_map.shard_dataset stream_map.shard_row_groups
Stream over row-groups/datasets and reducestream_reduce stream_reduce.shard_dataset stream_reduce.shard_row_groups
Stream sum of a numeric columnstream_sum
Stream top-k rows by a numeric columnstream_top_k
Check if shard_map Succeededsucceeded
Allocate a fixed-row table buffertable_buffer
Table Diagnosticstable_diagnostics
Finalize a table buffer or sinktable_finalize
Finalize a table buffertable_finalize.shard_table_buffer
Finalize a sinktable_finalize.shard_table_sink
Create a table sink for row-group or partitioned outputstable_sink
Write tabular results into a table buffer or sinktable_write
Write into a table buffertable_write.shard_table_buffer
Write a shard's row-group outputtable_write.shard_table_sink
Task Execution Reporttask_report
Utility Functionsutils
Create a view over a shared matrixview
Create a contiguous block viewview_block
View diagnosticsview_diagnostics
Create a gather (indexed) view over a shared matrixview_gather
Introspection for a viewview_info
Zero-copy Viewsviews
Individual Worker Controlworker