Skip to contents

The byterefs refs view of an array (dim_0.., present, path, offset, size, info, data), built in Rust as Arrow record batches and handed over as a nanoarrow_array_stream, without R vectors in between. path is dictionary-encoded (one entry per source), offset and size are int64, as the byterefs spec has them. Paths have the repository's remap rules undone (see ic_write_byterefs()); native chunks (bytes held by the repository) have no path, and their bytes in data when native = TRUE, as ic_to_byterefs() writes them.

Usage

ic_refs_arrow(session, path, chunks = NULL, batch_rows = 65536L, native = TRUE)

Arguments

session

an icechunk_session.

path

array path.

chunks

optional slab, list(dim_0 = c(lo, hi), ...) of chunk indices (hi exclusive); only the manifests covering it are read.

batch_rows

rows per record batch.

native

fetch native chunks' bytes into data (concurrently, see ic_manifest_concurrency()); FALSE leaves them null.

Value

a nanoarrow_array_stream.

Details

The stream is lazy: it reads the session as it is when called (later writes to the session are not seen), and builds each batch as the consumer asks for it, with one batch of read-ahead, so a large array's refs never sit in memory whole. Errors met while streaming (a manifest that cannot be fetched) are raised by the consumer.

Consume with nanoarrow::convert_array_stream(), arrow::as_arrow_table(), arrow::write_dataset() or duckdb::duckdb_register_arrow().

Examples

ex <- ic_example_repo()
s <- ic_session_readonly(ic_repo_open(ex$path, authorize = ex$authorize), branch = "main")
if (requireNamespace("nanoarrow", quietly = TRUE)) {
  st <- ic_refs_arrow(s, "from_bin", batch_rows = 2L)
  print(st$get_schema())
  # batches are built as they are pulled
  while (!is.null(b <- st$get_next())) print(nanoarrow::convert_array(b)[, c("dim_0", "offset", "size")])
  # one slab, with chunk bytes for inline/native chunks
  print(nanoarrow::convert_array_stream(ic_refs_arrow(s, "small", chunks = list(dim_0 = 1))))
}
#> <nanoarrow_schema struct>
#>  $ format    : chr "+s"
#>  $ name      : NULL
#>  $ metadata  : list()
#>  $ flags     : int 0
#>  $ children  :List of 8
#>   ..$ dim_0  :<nanoarrow_schema int32>
#>   .. ..$ format    : chr "i"
#>   .. ..$ name      : chr "dim_0"
#>   .. ..$ metadata  : list()
#>   .. ..$ flags     : int 0
#>   .. ..$ children  : list()
#>   .. ..$ dictionary: NULL
#>   ..$ dim_1  :<nanoarrow_schema int32>
#>   .. ..$ format    : chr "i"
#>   .. ..$ name      : chr "dim_1"
#>   .. ..$ metadata  : list()
#>   .. ..$ flags     : int 0
#>   .. ..$ children  : list()
#>   .. ..$ dictionary: NULL
#>   ..$ present:<nanoarrow_schema int32>
#>   .. ..$ format    : chr "i"
#>   .. ..$ name      : chr "present"
#>   .. ..$ metadata  : list()
#>   .. ..$ flags     : int 0
#>   .. ..$ children  : list()
#>   .. ..$ dictionary: NULL
#>   ..$ path   :<nanoarrow_schema dictionary(int32)<string>>
#>   .. ..$ format    : chr "i"
#>   .. ..$ name      : chr "path"
#>   .. ..$ metadata  : list()
#>   .. ..$ flags     : int 2
#>   .. ..$ children  : list()
#>   .. ..$ dictionary:<nanoarrow_schema string>
#>   .. .. ..$ format    : chr "u"
#>   .. .. ..$ name      : NULL
#>   .. .. ..$ metadata  : list()
#>   .. .. ..$ flags     : int 0
#>   .. .. ..$ children  : list()
#>   .. .. ..$ dictionary: NULL
#>   ..$ offset :<nanoarrow_schema int64>
#>   .. ..$ format    : chr "l"
#>   .. ..$ name      : chr "offset"
#>   .. ..$ metadata  : list()
#>   .. ..$ flags     : int 2
#>   .. ..$ children  : list()
#>   .. ..$ dictionary: NULL
#>   ..$ size   :<nanoarrow_schema int64>
#>   .. ..$ format    : chr "l"
#>   .. ..$ name      : chr "size"
#>   .. ..$ metadata  : list()
#>   .. ..$ flags     : int 2
#>   .. ..$ children  : list()
#>   .. ..$ dictionary: NULL
#>   ..$ info   :<nanoarrow_schema string>
#>   .. ..$ format    : chr "u"
#>   .. ..$ name      : chr "info"
#>   .. ..$ metadata  : list()
#>   .. ..$ flags     : int 2
#>   .. ..$ children  : list()
#>   .. ..$ dictionary: NULL
#>   ..$ data   :<nanoarrow_schema binary>
#>   .. ..$ format    : chr "z"
#>   .. ..$ name      : chr "data"
#>   .. ..$ metadata  : list()
#>   .. ..$ flags     : int 2
#>   .. ..$ children  : list()
#>   .. ..$ dictionary: NULL
#>  $ dictionary: NULL
#>   dim_0 offset size
#> 1     0      0 3072
#> 2     1   3072 3072
#>   dim_0 offset size
#> 1     2   6144 3072
#> 2     3   9216 3072
#>   dim_0 dim_1 present path offset size info       data
#> 1     1     0       1 <NA>     NA   NA <NA> blob[48 B]
#> 2     1     1       1 <NA>     NA   NA <NA> blob[48 B]