Skip to contents

Reads a byterefs table, either the SQLite working store (store.sqlite, or an open DBI connection) or the Parquet "flat" export (a directory holding arrays.json, sources.parquet and refs_<array>.parquet). byterefs/0.1 and its predecessor mdim-refs/0.2 are accepted.

Usage

ic_byterefs_read(x)

Arguments

x

path to a .sqlite file, a flat-export directory, a DBI connection, or a byterefs object (returned as is).

Value

a byterefs object: a list with format_version, meta (named list of strings), sources, arrays and remap data frames, and refs(array), a function returning that array's global refs rows (dim_0.., present, path, offset, size, info, data) joined to the source's etag, mtime and source_size. Read from SQLite, refs() queries the file lazily, so the connection stays open: call $close() when done (functions given a file path close it themselves).

Examples

ex <- ic_example_repo()
s <- ic_session_readonly(ic_repo_open(ex$path, authorize = ex$authorize), branch = "main")
# write the store's references as a byterefs SQLite file, read it back
if (requireNamespace("RSQLite", quietly = TRUE)) {
  db <- tempfile(fileext = ".sqlite")
  ic_to_byterefs(s, dsn = db)
  br <- ic_byterefs_read(db)
  print(br)
  print(br$arrays[, c("array", "kind", "shape", "chunks", "dtype")])
  print(head(br$refs("from_bin")))
  br$close()
}
#> <byterefs> byterefs/0.1 
#>   sources: 2
#>   from_bin             data  referenced <f4 64x48
#>   from_nc              data  referenced <f4 64x48
#>   small                data  referenced <f8 4x6
#>      array kind  shape chunks dtype
#> 1 from_bin data 64, 48 16, 48   <f4
#> 2  from_nc data 64, 48 64, 48   <f4
#> 3    small data   4, 6   2, 3   <f8
#>   dim_0 dim_1 present
#> 1     0     0       1
#> 2     1     0       1
#> 3     2     0       1
#> 4     3     0       1
#>                                                            path offset size
#> 1 /tmp/RtmpVaRScy/icechunk-example-35c971f6c977/source/grid.bin      0 3072
#> 2 /tmp/RtmpVaRScy/icechunk-example-35c971f6c977/source/grid.bin   3072 3072
#> 3 /tmp/RtmpVaRScy/icechunk-example-35c971f6c977/source/grid.bin   6144 3072
#> 4 /tmp/RtmpVaRScy/icechunk-example-35c971f6c977/source/grid.bin   9216 3072
#>   info data etag mtime source_size
#> 1 <NA> <NA> <NA>    NA          NA
#> 2 <NA> <NA> <NA>    NA          NA
#> 3 <NA> <NA> <NA>    NA          NA
#> 4 <NA> <NA> <NA>    NA          NA