Chunk references as tables: byterefs and writing through DBI
Source:vignettes/byterefs.Rmd
byterefs.RmdDraft. A first pass written to be rewritten. It runs offline on
ic_example_repo(), so it is checked with the package. Notes in italics mark where the real version should say more.
A virtual Icechunk array does not hold its data. It holds references: for each chunk, a file, a byte offset and a length. A table is the natural form for that information: one row per chunk, with columns for the chunk’s indices and where its bytes are. tabularberg works with references in that form both ways. It reads any snapshot out as tables and writes tables back in as commits.
The layout is byterefs (version 0.1), from raad.virtualize. Its tables:
| table | one row per | key columns |
|---|---|---|
meta |
key |
key, value
|
arrays |
array |
array, kind, storage,
dims, shape, chunks,
dtype, fill, codec,
attrs
|
sources |
source file |
source, size, mtime,
etag
|
refs_<array> |
chunk |
dim_0, dim_1, …, path,
offset, size, data
|
To add: why tables (SQL, Parquet, joins with file catalogues), and how this relates to kerchunk and VirtualiZarr manifests.
Out: a snapshot as tables
library(tabularberg)
ex <- ic_example_repo()
repo <- ic_repo_open(ex$path, authorize = ex$authorize)
s <- ic_session_readonly(repo, branch = "main")
br <- ic_to_byterefs(s)
br
#> <byterefs> byterefs/0.1
#> sources: 2
#> from_bin data referenced <f4 64x48
#> from_nc data referenced <f4 64x48
#> small data referenced <f8 4x6
br$arrays[, c("array", "kind", "storage", "shape", "chunks", "dtype")]
#> array kind storage shape chunks dtype
#> 1 from_bin data referenced 64, 48 16, 48 <f4
#> 2 from_nc data referenced 64, 48 64, 48 <f4
#> 3 small data referenced 4, 6 2, 3 <f8
br$refs("from_bin")[, c("dim_0", "dim_1", "path", "offset", "size")]
#> dim_0 dim_1 path offset size
#> 1 0 0 /tmp/RtmpqEnU7e/icechunk-example-381c1f977585/source/grid.bin 0 3072
#> 2 1 0 /tmp/RtmpqEnU7e/icechunk-example-381c1f977585/source/grid.bin 3072 3072
#> 3 2 0 /tmp/RtmpqEnU7e/icechunk-example-381c1f977585/source/grid.bin 6144 3072
#> 4 3 0 /tmp/RtmpqEnU7e/icechunk-example-381c1f977585/source/grid.bin 9216 3072from_bin and from_nc are virtual: their
rows point into files. The chunks of small were written
through a session and are small enough for Icechunk to keep inline, so
their bytes travel in the data column.
The same tables can be written to a file, a SQLite database that any SQL client can open:
db <- tempfile(fileext = ".sqlite")
ic_to_byterefs(s, dsn = db)
con <- DBI::dbConnect(RSQLite::SQLite(), db)
DBI::dbListTables(con)
#> [1] "arrays" "from_bin" "from_nc" "meta" "placements" "refs_from_bin"
#> [7] "refs_from_nc" "refs_small" "remap" "small" "sources"
DBI::dbGetQuery(con, "SELECT source, mtime FROM sources")
#> source mtime
#> 1 /tmp/RtmpqEnU7e/icechunk-example-381c1f977585/source/grid.bin NA
#> 2 /tmp/RtmpqEnU7e/icechunk-example-381c1f977585/source/grid.nc NA
DBI::dbDisconnect(con)In: tables as a repository
ic_repo_from_byterefs() goes the other way. It creates a
repository, declares the virtual chunk containers the paths need, writes
every array, and commits once.
br2 <- ic_byterefs_read(db)
copy <- ic_repo_from_byterefs(br2, tempfile("copy-"), authorize = ex$authorize,
message = "rebuilt from byterefs")
s2 <- ic_session_readonly(copy, branch = "main")
identical(ic_get(s2, "from_bin/c/2/0"), ic_get(s, "from_bin/c/2/0"))
#> [1] TRUE
cmp <- ic_byterefs_compare(br, ic_to_byterefs(s2))
all(cmp$ok)
#> [1] TRUE
br2$close() # a byterefs store read from SQLite holds the connection openWhen the sources table has a modification time (or an
ETag, for objects in a bucket), it is recorded as a checksum on that
source’s references. If the source file changes after the references
were made, Icechunk refuses to read its chunks rather than return the
wrong bytes. The example repository was made without checksums, so its
mtime column is empty.
To add: remap rules and targets, where the same references
resolve to a scratch path when building and to a public URL when
published (meta.default_target,
ic_byterefs_containers(br, target =)). This is the
raad.virtualize workflow and is better told from there.
Writing through DBI
The DBI connection is a way to read, and also a way to write.
Appending rows to arrays creates arrays, and appending rows
to refs_<array> sets chunk references. A transaction
becomes one Icechunk commit.
Start an empty repository that may read from the example’s source directory:
library(DBI)
path <- tempfile("built-")
con <- dbConnect(Icechunk(), path, create = TRUE,
containers = ic_container(ex$authorize, "local", path = ex$source),
authorize = ex$authorize)
dbListTables(con)
#> [1] "meta" "arrays" "sources" "branches" "tags" "ancestry" "nodes"Describe an array, then its chunks. grid.bin holds 64 x
48 little-endian float32 values. Here they become an array of 16-row
chunks:
dbBegin(con)
dbAppendTable(con, "arrays", data.frame(
array = "grid", kind = "data", storage = "referenced",
dims = '["y","x"]', shape = "[64,48]", chunks = "[16,48]", dtype = "<f4",
fill = "NaN", attrs = '{"units":"none"}'))
#> [1] 1
dbAppendTable(con, "refs_grid", data.frame(
dim_0 = 0:3, dim_1 = 0L,
path = file.path(ex$source, "grid.bin"), offset = (0:3) * 3072, size = 3072))
#> [1] 4
dbCommit(con, "grid: four blocks of grid.bin")
dbGetQuery(con, 'SELECT dim_0, "offset", size FROM refs_grid')
#> dim_0 offset size
#> 1 0 0 3072
#> 2 1 3072 3072
#> 3 2 6144 3072
#> 4 3 9216 3072The bytes come from the source file, through Icechunk:
rs <- ic_session_readonly(ic_repo_open(path, authorize = ex$authorize), branch = "main")
x <- readBin(ic_get(rs, "grid/c/3/0"), "double", 768, size = 4, endian = "little")
all.equal(x, ex$values[2304 + 1:768])
#> [1] TRUERows added later for a source already in the repository take that
source’s checksum, if it has one, from the sources table.
Without dbBegin(), each append commits on its own:
dbAppendTable(con, "refs_grid", data.frame(
dim_0 = 3L, dim_1 = 0L, path = file.path(ex$source, "grid.bin"), offset = 0, size = 3072))
#> [1] 1
dbGetQuery(con, "SELECT message FROM ancestry")
#> message
#> 1 DBI: 1 rows into refs_grid
#> 2 grid: four blocks of grid.bin
#> 3 Repository initializeddbRollback() discards an open transaction. If someone
else committed to the branch in the meantime,
dbCommit(rebase = TRUE) replays your changes on top of
theirs, with Icechunk’s conflict detection deciding what can merge.
other <- ic_session_writable(ic_repo_open(path, authorize = ex$authorize), "main")
ic_set(other, "zarr.json", charToRaw('{"zarr_format":3,"node_type":"group","attributes":{"by":"someone else"}}'))
ic_commit(other, "a concurrent edit")
#> [1] "G5XKX9ZZQ741RNVGRHS0"
dbBegin(con)
dbAppendTable(con, "refs_grid", data.frame(
dim_0 = 2L, dim_1 = 0L, path = file.path(ex$source, "grid.bin"), offset = 6144, size = 3072))
#> [1] 1
dbCommit(con, "restore block 2", rebase = TRUE)
dbGetQuery(con, "SELECT message FROM ancestry")
#> message
#> 1 restore block 2
#> 2 a concurrent edit
#> 3 DBI: 1 rows into refs_grid
#> 4 grid: four blocks of grid.bin
#> 5 Repository initialized
dbDisconnect(con)To add: building a catalogue from a file listing (GDAL / netCDF metadata to byterefs rows), and the size of real catalogues: the public OISST store has 16,482 references per variable, and GHRSST 935k.