Skip to contents

An Icechunk repository seen through DBI. Tables are the byterefs projection of the snapshot the connection is pinned to (see ic_to_byterefs()): meta, arrays, sources and one refs_<array> per data array, plus Icechunk's own branches, tags, ancestry and nodes. SQL (DBI::dbGetQuery()) runs on an in-memory SQLite copy of the tables a statement names: an in-memory SQLite copy (RSQLite), or with sql = "duckdb" a DuckDB registration that does not copy.

Usage

Icechunk()

# S4 method for class 'IcechunkDriver'
dbConnect(
  drv,
  storage,
  branch = NULL,
  tag = NULL,
  snapshot = NULL,
  authorize = NULL,
  containers = NULL,
  create = FALSE,
  target = NULL,
  sql = c("sqlite", "duckdb"),
  pushdown = TRUE,
  arrow = NA,
  ...
)

# S4 method for class 'IcechunkConnection'
dbListTables(conn, ...)

# S4 method for class 'IcechunkConnection,character'
dbExistsTable(conn, name, ...)

# S4 method for class 'IcechunkConnection,character'
dbReadTable(conn, name, ..., chunks = NULL)

# S4 method for class 'IcechunkConnection,character'
dbListFields(conn, name, ...)

# S4 method for class 'IcechunkConnection,character'
dbGetQuery(conn, statement, ...)

# S4 method for class 'IcechunkConnection,character'
dbSendQuery(conn, statement, ...)

# S4 method for class 'IcechunkResult'
dbFetch(res, n = -1, ...)

# S4 method for class 'IcechunkResult'
dbClearResult(res, ...)

# S4 method for class 'IcechunkResult'
dbHasCompleted(res, ...)

# S4 method for class 'IcechunkResult'
dbGetStatement(res, ...)

# S4 method for class 'IcechunkResult'
dbIsValid(dbObj, ...)

# S4 method for class 'IcechunkConnection,character'
dbExecute(conn, statement, ...)

# S4 method for class 'IcechunkConnection'
dbBegin(conn, ...)

# S4 method for class 'IcechunkConnection'
dbCommit(
  conn,
  message = "DBI commit",
  metadata = NULL,
  rebase = FALSE,
  solver = c("detect", "basic"),
  on_chunk = c("ours", "theirs", "fail"),
  tries = 5L,
  ...
)

# S4 method for class 'IcechunkConnection'
dbRollback(conn, ...)

# S4 method for class 'IcechunkConnection'
dbAppendTable(conn, name, value, ..., checksums = TRUE, row.names = NULL)

# S4 method for class 'IcechunkConnection,character,data.frame'
dbWriteTable(conn, name, value, ..., overwrite = FALSE, append = FALSE)

# S4 method for class 'IcechunkConnection'
dbDisconnect(conn, ...)

# S4 method for class 'IcechunkConnection'
dbIsValid(dbObj, ...)

# S4 method for class 'IcechunkDriver'
dbIsValid(dbObj, ...)

# S4 method for class 'IcechunkConnection'
dbGetInfo(dbObj, ...)

# S4 method for class 'IcechunkDriver'
dbGetInfo(dbObj, ...)

# S4 method for class 'IcechunkConnection'
show(object)

# S4 method for class 'IcechunkDriver'
show(object)

# S4 method for class 'IcechunkDriver'
dbDataType(dbObj, obj, ...)

Arguments

drv

an IcechunkDriver.

storage

an ic_storage object or a local path.

branch, tag, snapshot

what the connection is pinned to (one of them; default branch "main"). Only a branch connection can write.

authorize, containers

as for ic_repo_open().

create

create the repository if it does not exist.

target

remap target used to resolve path on writes (default: the one recorded by ic_write_byterefs(), if any).

sql

SQL engine for DBI::dbGetQuery(): "sqlite" (default) copies each table a statement names into an in-memory SQLite; "duckdb" registers them with an in-memory DuckDB without copying, faster on large refs tables. In DuckDB offset and array are reserved words, so quote those columns ("offset", "array"); SQL written that way runs on both engines.

pushdown

for a statement with a plain conjunctive WHERE (no OR, NOT, CASE, subquery or set operation) on dim_k (=, <, <=, >, >=, BETWEEN, IN), read only that slab of chunks of each refs_<array> table it names (and, with split manifests, only the manifests covering it) instead of the whole table. In joins, a predicate counts for a table when qualified by its name or alias, and ranges carry across a.dim_i = b.dim_j equalities, so a filter on one side narrows the other; self-joins and NATURAL / USING joins are read whole. The SQL still applies the full predicate, so results do not change.

arrow

with sql = "duckdb", hand refs_<array> tables to DuckDB as Arrow record batches built in Rust (ic_refs_arrow(), native chunks' bytes included) rather than R vectors: less memory, faster first query, same results. NA (default): when nanoarrow and arrow are installed.

...

unused.

conn

an IcechunkConnection.

name

table name.

chunks

for dbReadTable() on refs_<array>: a slab of chunk indices, list(dim_0 = c(lo, hi), ...) (hi exclusive; a single number is one index). Only the manifests covering it are read when the repository splits manifests (see manifest_split in ic_repo_create()).

statement

SQL (read-only: SELECT over the tables it names).

res

an IcechunkResult.

n

rows to fetch (-1 for all).

dbObj

an Icechunk DBI object.

message

commit message.

metadata

commit metadata (named list).

rebase

if another writer moved the branch since DBI::dbBegin(), rebase onto the new head and retry, up to tries times. With solver = "detect" this succeeds only when the two change sets touch different chunks and metadata; solver = "basic" also resolves chunk conflicts, keeping on_chunk ("ours" or "theirs"). An unresolvable conflict is signalled as icechunk_conflict and leaves the transaction open (rebased as far as it got), for DBI::dbRollback() or another try.

solver, on_chunk

as for ic_rebase().

tries

maximum commit attempts with rebase = TRUE.

value

a data frame. For refs_<array>: dim_0.., and either path (+ offset, size) for a virtual ref or data (raw) for an inline chunk; optional present (0 rows are skipped), etag, mtime. For arrays: byterefs arrays rows (JSON text columns), creating each array.

checksums

record etag / mtime checksums on virtual refs. Rows without them (columns absent or NA) take the checksum of the same path in the sources table, so refs to a known source are checked like the rest; a path new to the repository gets none unless supplied.

row.names

ignored (required by the DBI generic).

overwrite, append

only appending is supported.

object

an Icechunk DBI object.

obj

an R object.

Value

Icechunk(): an IcechunkDriver.

Details

Writes go through a writable session on the connection's branch: DBI::dbBegin() opens it, DBI::dbAppendTable() on refs_<array> sets virtual (or inline, from data) chunk refs, on arrays creates arrays, DBI::dbCommit() commits (pass message), DBI::dbRollback() discards. Without dbBegin() each append commits on its own.

Examples

if (requireNamespace("RSQLite", quietly = TRUE)) {
  library(DBI)
  ex <- ic_example_repo()
  con <- dbConnect(Icechunk(), ex$path, authorize = ex$authorize)
  print(con)
  print(dbListTables(con))
  print(dbGetQuery(con, 'SELECT "array", kind, shape, chunks, dtype FROM arrays'))
  # chunk references: one row per chunk; WHERE on dim_k reads only that slab
  print(dbGetQuery(con, 'SELECT dim_0, path, "offset", size FROM refs_from_bin WHERE dim_0 >= 2'))
  print(dbReadTable(con, "ancestry")[, c("snapshot_id", "message")])

  # write: append references in a transaction, which becomes one commit
  dbBegin(con)
  dbAppendTable(con, "refs_from_bin", data.frame(
    dim_0 = 3L, dim_1 = 0L, path = file.path(ex$source, "grid.bin"), offset = 0, size = 3072))
  dbCommit(con, "point the last block at the first")
  print(dbGetQuery(con, 'SELECT dim_0, "offset" FROM refs_from_bin'))

  # an earlier state, read-only
  old <- dbConnect(Icechunk(), ex$path, tag = "v1", authorize = ex$authorize)
  print(dbListTables(old))
  dbDisconnect(old); dbDisconnect(con)
}
#> <IcechunkConnection> branch main @ DKNW2M4ZRXS5344EEXCG
#>  [1] "meta"          "arrays"        "sources"       "branches"     
#>  [5] "tags"          "ancestry"      "nodes"         "refs_from_bin"
#>  [9] "refs_from_nc"  "refs_small"   
#>      array kind   shape  chunks dtype
#> 1 from_bin data [64,48] [16,48]   <f4
#> 2  from_nc data [64,48] [64,48]   <f4
#> 3    small data   [4,6]   [2,3]   <f8
#>   dim_0                                                         path offset
#> 1     2 /tmp/RtmpVaRScy/icechunk-example-35c9a0dce32/source/grid.bin   6144
#> 2     3 /tmp/RtmpVaRScy/icechunk-example-35c9a0dce32/source/grid.bin   9216
#>   size
#> 1 3072
#> 2 3072
#>            snapshot_id                          message
#> 1 DKNW2M4ZRXS5344EEXCG                    a small array
#> 2 3NWPYAGTJQS3G4PH3PSG two arrays of virtual references
#> 3 1CECHNKREP0F1RSTCMT0           Repository initialized
#>   dim_0 offset
#> 1     0      0
#> 2     1   3072
#> 3     2   6144
#> 4     3      0
#> [1] "meta"          "arrays"        "sources"       "branches"     
#> [5] "tags"          "ancestry"      "nodes"         "refs_from_bin"
#> [9] "refs_from_nc"