Skip to contents

Draft. A first pass written to be rewritten. It runs offline on ic_example_repo(), so it is checked with the package. Notes in italics mark where the real version should say more.

A virtual Icechunk array does not hold its data. It holds references: for each chunk, a file, a byte offset and a length. A table is the natural form for that information: one row per chunk, with columns for the chunk’s indices and where its bytes are. tabularberg works with references in that form both ways. It reads any snapshot out as tables and writes tables back in as commits.

The layout is byterefs (version 0.1), from raad.virtualize. Its tables:

table one row per key columns
meta key key, value
arrays array array, kind, storage, dims, shape, chunks, dtype, fill, codec, attrs
sources source file source, size, mtime, etag
refs_<array> chunk dim_0, dim_1, …, path, offset, size, data

To add: why tables (SQL, Parquet, joins with file catalogues), and how this relates to kerchunk and VirtualiZarr manifests.

Out: a snapshot as tables

library(tabularberg)
ex <- ic_example_repo()
repo <- ic_repo_open(ex$path, authorize = ex$authorize)
s <- ic_session_readonly(repo, branch = "main")

br <- ic_to_byterefs(s)
br
#> <byterefs> byterefs/0.1 
#>   sources: 2
#>   from_bin             data  referenced <f4 64x48
#>   from_nc              data  referenced <f4 64x48
#>   small                data  referenced <f8 4x6
br$arrays[, c("array", "kind", "storage", "shape", "chunks", "dtype")]
#>      array kind    storage  shape chunks dtype
#> 1 from_bin data referenced 64, 48 16, 48   <f4
#> 2  from_nc data referenced 64, 48 64, 48   <f4
#> 3    small data referenced   4, 6   2, 3   <f8
br$refs("from_bin")[, c("dim_0", "dim_1", "path", "offset", "size")]
#>   dim_0 dim_1                                                          path offset size
#> 1     0     0 /tmp/RtmpqEnU7e/icechunk-example-381c1f977585/source/grid.bin      0 3072
#> 2     1     0 /tmp/RtmpqEnU7e/icechunk-example-381c1f977585/source/grid.bin   3072 3072
#> 3     2     0 /tmp/RtmpqEnU7e/icechunk-example-381c1f977585/source/grid.bin   6144 3072
#> 4     3     0 /tmp/RtmpqEnU7e/icechunk-example-381c1f977585/source/grid.bin   9216 3072

from_bin and from_nc are virtual: their rows point into files. The chunks of small were written through a session and are small enough for Icechunk to keep inline, so their bytes travel in the data column.

The same tables can be written to a file, a SQLite database that any SQL client can open:

db <- tempfile(fileext = ".sqlite")
ic_to_byterefs(s, dsn = db)
con <- DBI::dbConnect(RSQLite::SQLite(), db)
DBI::dbListTables(con)
#>  [1] "arrays"        "from_bin"      "from_nc"       "meta"          "placements"    "refs_from_bin"
#>  [7] "refs_from_nc"  "refs_small"    "remap"         "small"         "sources"
DBI::dbGetQuery(con, "SELECT source, mtime FROM sources")
#>                                                          source mtime
#> 1 /tmp/RtmpqEnU7e/icechunk-example-381c1f977585/source/grid.bin    NA
#> 2  /tmp/RtmpqEnU7e/icechunk-example-381c1f977585/source/grid.nc    NA
DBI::dbDisconnect(con)

In: tables as a repository

ic_repo_from_byterefs() goes the other way. It creates a repository, declares the virtual chunk containers the paths need, writes every array, and commits once.

br2 <- ic_byterefs_read(db)
copy <- ic_repo_from_byterefs(br2, tempfile("copy-"), authorize = ex$authorize,
                              message = "rebuilt from byterefs")
s2 <- ic_session_readonly(copy, branch = "main")
identical(ic_get(s2, "from_bin/c/2/0"), ic_get(s, "from_bin/c/2/0"))
#> [1] TRUE
cmp <- ic_byterefs_compare(br, ic_to_byterefs(s2))
all(cmp$ok)
#> [1] TRUE
br2$close()   # a byterefs store read from SQLite holds the connection open

When the sources table has a modification time (or an ETag, for objects in a bucket), it is recorded as a checksum on that source’s references. If the source file changes after the references were made, Icechunk refuses to read its chunks rather than return the wrong bytes. The example repository was made without checksums, so its mtime column is empty.

To add: remap rules and targets, where the same references resolve to a scratch path when building and to a public URL when published (meta.default_target, ic_byterefs_containers(br, target =)). This is the raad.virtualize workflow and is better told from there.

Writing through DBI

The DBI connection is a way to read, and also a way to write. Appending rows to arrays creates arrays, and appending rows to refs_<array> sets chunk references. A transaction becomes one Icechunk commit.

Start an empty repository that may read from the example’s source directory:

library(DBI)
path <- tempfile("built-")
con <- dbConnect(Icechunk(), path, create = TRUE,
                 containers = ic_container(ex$authorize, "local", path = ex$source),
                 authorize = ex$authorize)
dbListTables(con)
#> [1] "meta"     "arrays"   "sources"  "branches" "tags"     "ancestry" "nodes"

Describe an array, then its chunks. grid.bin holds 64 x 48 little-endian float32 values. Here they become an array of 16-row chunks:

dbBegin(con)
dbAppendTable(con, "arrays", data.frame(
  array = "grid", kind = "data", storage = "referenced",
  dims = '["y","x"]', shape = "[64,48]", chunks = "[16,48]", dtype = "<f4",
  fill = "NaN", attrs = '{"units":"none"}'))
#> [1] 1
dbAppendTable(con, "refs_grid", data.frame(
  dim_0 = 0:3, dim_1 = 0L,
  path = file.path(ex$source, "grid.bin"), offset = (0:3) * 3072, size = 3072))
#> [1] 4
dbCommit(con, "grid: four blocks of grid.bin")
dbGetQuery(con, 'SELECT dim_0, "offset", size FROM refs_grid')
#>   dim_0 offset size
#> 1     0      0 3072
#> 2     1   3072 3072
#> 3     2   6144 3072
#> 4     3   9216 3072

The bytes come from the source file, through Icechunk:

rs <- ic_session_readonly(ic_repo_open(path, authorize = ex$authorize), branch = "main")
x <- readBin(ic_get(rs, "grid/c/3/0"), "double", 768, size = 4, endian = "little")
all.equal(x, ex$values[2304 + 1:768])
#> [1] TRUE

Rows added later for a source already in the repository take that source’s checksum, if it has one, from the sources table. Without dbBegin(), each append commits on its own:

dbAppendTable(con, "refs_grid", data.frame(
  dim_0 = 3L, dim_1 = 0L, path = file.path(ex$source, "grid.bin"), offset = 0, size = 3072))
#> [1] 1
dbGetQuery(con, "SELECT message FROM ancestry")
#>                         message
#> 1    DBI: 1 rows into refs_grid
#> 2 grid: four blocks of grid.bin
#> 3        Repository initialized

dbRollback() discards an open transaction. If someone else committed to the branch in the meantime, dbCommit(rebase = TRUE) replays your changes on top of theirs, with Icechunk’s conflict detection deciding what can merge.

other <- ic_session_writable(ic_repo_open(path, authorize = ex$authorize), "main")
ic_set(other, "zarr.json", charToRaw('{"zarr_format":3,"node_type":"group","attributes":{"by":"someone else"}}'))
ic_commit(other, "a concurrent edit")
#> [1] "G5XKX9ZZQ741RNVGRHS0"

dbBegin(con)
dbAppendTable(con, "refs_grid", data.frame(
  dim_0 = 2L, dim_1 = 0L, path = file.path(ex$source, "grid.bin"), offset = 6144, size = 3072))
#> [1] 1
dbCommit(con, "restore block 2", rebase = TRUE)
dbGetQuery(con, "SELECT message FROM ancestry")
#>                         message
#> 1               restore block 2
#> 2             a concurrent edit
#> 3    DBI: 1 rows into refs_grid
#> 4 grid: four blocks of grid.bin
#> 5        Repository initialized
dbDisconnect(con)

To add: building a catalogue from a file listing (GDAL / netCDF metadata to byterefs rows), and the size of real catalogues: the public OISST store has 16,482 references per variable, and GHRSST 935k.