BRAN2023 through kerchunk-Parquet references

kerchunk

Parquet

Zarr

THREDDS

ocean

A 13 TB ocean reanalysis on NCI THREDDS, read through a few megabytes of Parquet references.

BRAN2023, the Bluelink ocean reanalysis (CSIRO and the Bureau of Meteorology), is daily, 0.1 degree, 51 levels, from 1993: hundreds of terabytes of netCDF on NCI’s THREDDS server. kerchunk references in Parquet describe every chunk of every file, and GDAL’s Zarr driver reads them as one store. See Byte-reference stores for what such a reference table holds.

THREDDS limits aggressive parallel reads. Eight threads is a safe cap:

dask.config.set(scheduler="threads", num_workers=8)

<dask.config.set at 0x7fc410434050>

A pinned reference store

Live: BRAN2023 ocean temp as kerchunk-Parquet Zarr (GitHub, pinned commit)

Address: ZARR:"/vsicurl/https://raw.githubusercontent.com/mdsumner/virtualized/d0b9c23317d2de07c3851eb9dfc968be1761d5b8/zarr/ocean_temp.zarr"
Provider: mdsumner/virtualized at d0b9c23; chunks on NCI THREDDS
Pinned: a fixed commit or immutable object, will not change.
Status: answered the store check at 2026-11-01 09:15 UTC.

dsn = ('ZARR:"/vsicurl/https://raw.githubusercontent.com/mdsumner/virtualized/'
       'd0b9c23317d2de07c3851eb9dfc968be1761d5b8/zarr/ocean_temp.zarr"')
if live("bran-virtualized-pinned"):
    bran = xr.open_dataset(dsn, engine="gdalxarray", drop_variables=["Time_bnds"])
    print(bran)

<xarray.Dataset> Size: 13TB
Dimensions:   (Time: 5844, st_ocean: 51, yt_ocean: 1500, xt_ocean: 3600, nv: 2)
Coordinates:
  * Time      (Time) float64 47kB 1.132e+04 1.132e+04 ... 1.717e+04 1.717e+04
  * st_ocean  (st_ocean) float64 408B 2.5 7.5 12.5 ... 3.603e+03 4.509e+03
  * yt_ocean  (yt_ocean) float64 12kB -74.95 -74.85 -74.75 ... 74.75 74.85 74.95
  * xt_ocean  (xt_ocean) float64 29kB 0.05 0.15 0.25 0.35 ... 359.8 359.9 360.0
  * nv        (nv) float64 16B 1.0 2.0
Data variables:
    temp      (Time, st_ocean, yt_ocean, xt_ocean) float64 13TB ...

Time comes through as numbers here: the references carry the values and units of the source, and decoding them is a one-liner if they are not decoded already. One day’s surface temperature over the Tasman Sea, chunked so dask reads the few chunks in parallel:

if live("bran-virtualized-pinned"):
    surf = (bran["temp"].isel(Time=0, st_ocean=0)
            .sel(yt_ocean=slice(-48, -25), xt_ocean=slice(145, 170))
            .chunk({"yt_ocean": 300, "xt_ocean": 300}).load())
    fig, ax = plt.subplots(figsize=(7, 6))
    surf.plot(ax=ax, cmap="RdYlBu_r")
    ax.set_aspect(1 / np.cos(np.radians(-36)))
    plt.show()

BRAN2023 surface temperature, first day of the store, read through kerchunk references from THREDDS.

Yearly reference stores on Pawsey

The same references, one store per variable per year, hosted on Pawsey’s object store and opened through a vrt:// wrapper:

Not run in this build: BRAN2023 ocean temp 2023 kerchunk-Parquet (Pawsey aad-index)

Address: vrt:///vsicurl/https://projects.pawsey.org.au/aad-index/vzarr/ocean_temp_2023.parq
Provider: AAD on Pawsey Acacia
Not yet classified.
Status: the store did not answer at 2026-11-01 09:15 UTC: HTTP response code: 404

dsn = "vrt:///vsicurl/https://projects.pawsey.org.au/aad-index/vzarr/ocean_temp_2023.parq"
if live("bran-pawsey-2023"):
    b23 = xr.open_dataset(dsn, engine="gdalxarray",
                          drop_variables=["Time_bnds", "DT", "average_DT", "average_T1", "average_T2"])
    print(dict(b23.sizes))

Rendered with GDAL 3.13.3, xarray 2026.4.1.dev32+g71cfb13ac, numpy 2.5.4, gdalxarray 0.5.1, rasterix 0.2.4.