The byterefs refs view of an array (dim_0.., present, path,
offset, size, info, data), built in Rust as Arrow record batches
and handed over as a nanoarrow_array_stream, without R vectors in
between. path is dictionary-encoded (one entry per source), offset
and size are int64, as the byterefs spec has them. Paths have the
repository's remap rules undone (see ic_write_byterefs()); native
chunks (bytes held by the repository) have no path, and their bytes in
data when native = TRUE, as ic_to_byterefs() writes them.
Arguments
- session
an
icechunk_session.- path
array path.
- chunks
optional slab,
list(dim_0 = c(lo, hi), ...)of chunk indices (hi exclusive); only the manifests covering it are read.- batch_rows
rows per record batch.
- native
fetch native chunks' bytes into
data(concurrently, seeic_manifest_concurrency());FALSEleaves them null.
Details
The stream is lazy: it reads the session as it is when called (later writes to the session are not seen), and builds each batch as the consumer asks for it, with one batch of read-ahead, so a large array's refs never sit in memory whole. Errors met while streaming (a manifest that cannot be fetched) are raised by the consumer.
Consume with nanoarrow::convert_array_stream(), arrow::as_arrow_table(),
arrow::write_dataset() or duckdb::duckdb_register_arrow().
Examples
ex <- ic_example_repo()
s <- ic_session_readonly(ic_repo_open(ex$path, authorize = ex$authorize), branch = "main")
if (requireNamespace("nanoarrow", quietly = TRUE)) {
st <- ic_refs_arrow(s, "from_bin", batch_rows = 2L)
print(st$get_schema())
# batches are built as they are pulled
while (!is.null(b <- st$get_next())) print(nanoarrow::convert_array(b)[, c("dim_0", "offset", "size")])
# one slab, with chunk bytes for inline/native chunks
print(nanoarrow::convert_array_stream(ic_refs_arrow(s, "small", chunks = list(dim_0 = 1))))
}
#> <nanoarrow_schema struct>
#> $ format : chr "+s"
#> $ name : NULL
#> $ metadata : list()
#> $ flags : int 0
#> $ children :List of 8
#> ..$ dim_0 :<nanoarrow_schema int32>
#> .. ..$ format : chr "i"
#> .. ..$ name : chr "dim_0"
#> .. ..$ metadata : list()
#> .. ..$ flags : int 0
#> .. ..$ children : list()
#> .. ..$ dictionary: NULL
#> ..$ dim_1 :<nanoarrow_schema int32>
#> .. ..$ format : chr "i"
#> .. ..$ name : chr "dim_1"
#> .. ..$ metadata : list()
#> .. ..$ flags : int 0
#> .. ..$ children : list()
#> .. ..$ dictionary: NULL
#> ..$ present:<nanoarrow_schema int32>
#> .. ..$ format : chr "i"
#> .. ..$ name : chr "present"
#> .. ..$ metadata : list()
#> .. ..$ flags : int 0
#> .. ..$ children : list()
#> .. ..$ dictionary: NULL
#> ..$ path :<nanoarrow_schema dictionary(int32)<string>>
#> .. ..$ format : chr "i"
#> .. ..$ name : chr "path"
#> .. ..$ metadata : list()
#> .. ..$ flags : int 2
#> .. ..$ children : list()
#> .. ..$ dictionary:<nanoarrow_schema string>
#> .. .. ..$ format : chr "u"
#> .. .. ..$ name : NULL
#> .. .. ..$ metadata : list()
#> .. .. ..$ flags : int 0
#> .. .. ..$ children : list()
#> .. .. ..$ dictionary: NULL
#> ..$ offset :<nanoarrow_schema int64>
#> .. ..$ format : chr "l"
#> .. ..$ name : chr "offset"
#> .. ..$ metadata : list()
#> .. ..$ flags : int 2
#> .. ..$ children : list()
#> .. ..$ dictionary: NULL
#> ..$ size :<nanoarrow_schema int64>
#> .. ..$ format : chr "l"
#> .. ..$ name : chr "size"
#> .. ..$ metadata : list()
#> .. ..$ flags : int 2
#> .. ..$ children : list()
#> .. ..$ dictionary: NULL
#> ..$ info :<nanoarrow_schema string>
#> .. ..$ format : chr "u"
#> .. ..$ name : chr "info"
#> .. ..$ metadata : list()
#> .. ..$ flags : int 2
#> .. ..$ children : list()
#> .. ..$ dictionary: NULL
#> ..$ data :<nanoarrow_schema binary>
#> .. ..$ format : chr "z"
#> .. ..$ name : chr "data"
#> .. ..$ metadata : list()
#> .. ..$ flags : int 2
#> .. ..$ children : list()
#> .. ..$ dictionary: NULL
#> $ dictionary: NULL
#> dim_0 offset size
#> 1 0 0 3072
#> 2 1 3072 3072
#> dim_0 offset size
#> 1 2 6144 3072
#> 2 3 9216 3072
#> dim_0 dim_1 present path offset size info data
#> 1 1 0 1 <NA> NA NA <NA> blob[48 B]
#> 2 1 1 1 <NA> NA NA <NA> blob[48 B]