Lazy data in a Variable

A Variable's data can be a lazy chunked array from the altarr package: an ordinary double, integer or logical vector with a dim attribute that reads its values chunk by chunk only when they are asked for. Variables read by open_dataset() hold such arrays, and any altarr array can be used directly:

Details

v <- Variable(dims = c("lon", "lat", "time"), data = altarr::altarr(...))

Every Variable, DataArray and Dataset method accepts lazy data:

• dim(), shape(), printing and coordinate work read nothing.

• isel() and sel() read nothing: they return a Variable whose data is a lazy view of the selection, still backed by the same chunks.

• Reductions (nd_mean() and friends) stream the array in blocks aligned to its chunks, one planned read per block, and never hold more than one block plus the result in memory.

• Arithmetic, as.array(), as.data.frame() and collect() read the (selected) values into memory with one planned read.

The block size for reductions is getOption("ndr.block_values") values (default 2^22); the chunk cache and fetch batching are altarr's options (see ?altarr::altarr_contract).