An Icechunk repository seen through DBI. Tables are the byterefs
projection of the snapshot the connection is pinned to (see
ic_to_byterefs()): meta, arrays, sources and one refs_<array>
per data array, plus Icechunk's own branches, tags, ancestry and
nodes. SQL (DBI::dbGetQuery()) runs on an in-memory SQLite copy of
the tables a statement names: an in-memory SQLite copy (RSQLite), or with
sql = "duckdb" a DuckDB registration that does not copy.
Usage
Icechunk()
# S4 method for class 'IcechunkDriver'
dbConnect(
drv,
storage,
branch = NULL,
tag = NULL,
snapshot = NULL,
authorize = NULL,
containers = NULL,
create = FALSE,
target = NULL,
sql = c("sqlite", "duckdb"),
pushdown = TRUE,
arrow = NA,
...
)
# S4 method for class 'IcechunkConnection'
dbListTables(conn, ...)
# S4 method for class 'IcechunkConnection,character'
dbExistsTable(conn, name, ...)
# S4 method for class 'IcechunkConnection,character'
dbReadTable(conn, name, ..., chunks = NULL)
# S4 method for class 'IcechunkConnection,character'
dbListFields(conn, name, ...)
# S4 method for class 'IcechunkConnection,character'
dbGetQuery(conn, statement, ...)
# S4 method for class 'IcechunkConnection,character'
dbSendQuery(conn, statement, ...)
# S4 method for class 'IcechunkResult'
dbFetch(res, n = -1, ...)
# S4 method for class 'IcechunkResult'
dbClearResult(res, ...)
# S4 method for class 'IcechunkResult'
dbHasCompleted(res, ...)
# S4 method for class 'IcechunkResult'
dbGetStatement(res, ...)
# S4 method for class 'IcechunkResult'
dbIsValid(dbObj, ...)
# S4 method for class 'IcechunkConnection,character'
dbExecute(conn, statement, ...)
# S4 method for class 'IcechunkConnection'
dbBegin(conn, ...)
# S4 method for class 'IcechunkConnection'
dbCommit(
conn,
message = "DBI commit",
metadata = NULL,
rebase = FALSE,
solver = c("detect", "basic"),
on_chunk = c("ours", "theirs", "fail"),
tries = 5L,
...
)
# S4 method for class 'IcechunkConnection'
dbRollback(conn, ...)
# S4 method for class 'IcechunkConnection'
dbAppendTable(conn, name, value, ..., checksums = TRUE, row.names = NULL)
# S4 method for class 'IcechunkConnection,character,data.frame'
dbWriteTable(conn, name, value, ..., overwrite = FALSE, append = FALSE)
# S4 method for class 'IcechunkConnection'
dbDisconnect(conn, ...)
# S4 method for class 'IcechunkConnection'
dbIsValid(dbObj, ...)
# S4 method for class 'IcechunkDriver'
dbIsValid(dbObj, ...)
# S4 method for class 'IcechunkConnection'
dbGetInfo(dbObj, ...)
# S4 method for class 'IcechunkDriver'
dbGetInfo(dbObj, ...)
# S4 method for class 'IcechunkConnection'
show(object)
# S4 method for class 'IcechunkDriver'
show(object)
# S4 method for class 'IcechunkDriver'
dbDataType(dbObj, obj, ...)Arguments
- drv
an
IcechunkDriver.- storage
an ic_storage object or a local path.
- branch, tag, snapshot
what the connection is pinned to (one of them; default branch
"main"). Only a branch connection can write.as for
ic_repo_open().- create
create the repository if it does not exist.
- target
remap target used to resolve
pathon writes (default: the one recorded byic_write_byterefs(), if any).- sql
SQL engine for
DBI::dbGetQuery():"sqlite"(default) copies each table a statement names into an in-memory SQLite;"duckdb"registers them with an in-memory DuckDB without copying, faster on large refs tables. In DuckDBoffsetandarrayare reserved words, so quote those columns ("offset","array"); SQL written that way runs on both engines.- pushdown
for a statement with a plain conjunctive
WHERE(noOR,NOT,CASE, subquery or set operation) ondim_k(=,<,<=,>,>=,BETWEEN,IN), read only that slab of chunks of eachrefs_<array>table it names (and, with split manifests, only the manifests covering it) instead of the whole table. In joins, a predicate counts for a table when qualified by its name or alias, and ranges carry acrossa.dim_i = b.dim_jequalities, so a filter on one side narrows the other; self-joins andNATURAL/USINGjoins are read whole. The SQL still applies the full predicate, so results do not change.- arrow
with
sql = "duckdb", handrefs_<array>tables to DuckDB as Arrow record batches built in Rust (ic_refs_arrow(), native chunks' bytes included) rather than R vectors: less memory, faster first query, same results.NA(default): when nanoarrow and arrow are installed.- ...
unused.
- conn
an
IcechunkConnection.- name
table name.
- chunks
for
dbReadTable()onrefs_<array>: a slab of chunk indices,list(dim_0 = c(lo, hi), ...)(hi exclusive; a single number is one index). Only the manifests covering it are read when the repository splits manifests (seemanifest_splitinic_repo_create()).- statement
SQL (read-only: SELECT over the tables it names).
- res
an
IcechunkResult.- n
rows to fetch (-1 for all).
- dbObj
an Icechunk DBI object.
- message
commit message.
- metadata
commit metadata (named list).
- rebase
if another writer moved the branch since
DBI::dbBegin(), rebase onto the new head and retry, up totriestimes. Withsolver = "detect"this succeeds only when the two change sets touch different chunks and metadata;solver = "basic"also resolves chunk conflicts, keepingon_chunk("ours"or"theirs"). An unresolvable conflict is signalled asicechunk_conflictand leaves the transaction open (rebased as far as it got), forDBI::dbRollback()or another try.- solver, on_chunk
as for
ic_rebase().- tries
maximum commit attempts with
rebase = TRUE.- value
a data frame. For
refs_<array>:dim_0.., and eitherpath(+offset,size) for a virtual ref ordata(raw) for an inline chunk; optionalpresent(0 rows are skipped),etag,mtime. Forarrays: byterefsarraysrows (JSON text columns), creating each array.- checksums
record
etag/mtimechecksums on virtual refs. Rows without them (columns absent orNA) take the checksum of the samepathin thesourcestable, so refs to a known source are checked like the rest; a path new to the repository gets none unless supplied.- row.names
ignored (required by the DBI generic).
- overwrite, append
only appending is supported.
- object
an Icechunk DBI object.
- obj
an R object.
Details
Writes go through a writable session on the connection's branch:
DBI::dbBegin() opens it, DBI::dbAppendTable() on refs_<array> sets
virtual (or inline, from data) chunk refs, on arrays creates arrays,
DBI::dbCommit() commits (pass message), DBI::dbRollback()
discards. Without dbBegin() each append commits on its own.
Examples
if (requireNamespace("RSQLite", quietly = TRUE)) {
library(DBI)
ex <- ic_example_repo()
con <- dbConnect(Icechunk(), ex$path, authorize = ex$authorize)
print(con)
print(dbListTables(con))
print(dbGetQuery(con, 'SELECT "array", kind, shape, chunks, dtype FROM arrays'))
# chunk references: one row per chunk; WHERE on dim_k reads only that slab
print(dbGetQuery(con, 'SELECT dim_0, path, "offset", size FROM refs_from_bin WHERE dim_0 >= 2'))
print(dbReadTable(con, "ancestry")[, c("snapshot_id", "message")])
# write: append references in a transaction, which becomes one commit
dbBegin(con)
dbAppendTable(con, "refs_from_bin", data.frame(
dim_0 = 3L, dim_1 = 0L, path = file.path(ex$source, "grid.bin"), offset = 0, size = 3072))
dbCommit(con, "point the last block at the first")
print(dbGetQuery(con, 'SELECT dim_0, "offset" FROM refs_from_bin'))
# an earlier state, read-only
old <- dbConnect(Icechunk(), ex$path, tag = "v1", authorize = ex$authorize)
print(dbListTables(old))
dbDisconnect(old); dbDisconnect(con)
}
#> <IcechunkConnection> branch main @ DKNW2M4ZRXS5344EEXCG
#> [1] "meta" "arrays" "sources" "branches"
#> [5] "tags" "ancestry" "nodes" "refs_from_bin"
#> [9] "refs_from_nc" "refs_small"
#> array kind shape chunks dtype
#> 1 from_bin data [64,48] [16,48] <f4
#> 2 from_nc data [64,48] [64,48] <f4
#> 3 small data [4,6] [2,3] <f8
#> dim_0 path offset
#> 1 2 /tmp/RtmpVaRScy/icechunk-example-35c9a0dce32/source/grid.bin 6144
#> 2 3 /tmp/RtmpVaRScy/icechunk-example-35c9a0dce32/source/grid.bin 9216
#> size
#> 1 3072
#> 2 3072
#> snapshot_id message
#> 1 DKNW2M4ZRXS5344EEXCG a small array
#> 2 3NWPYAGTJQS3G4PH3PSG two arrays of virtual references
#> 3 1CECHNKREP0F1RSTCMT0 Repository initialized
#> dim_0 offset
#> 1 0 0
#> 2 1 3072
#> 3 2 6144
#> 4 3 0
#> [1] "meta" "arrays" "sources" "branches"
#> [5] "tags" "ancestry" "nodes" "refs_from_bin"
#> [9] "refs_from_nc"