Skip to contents

cmrlifecycle

Small package for auditing the update/revision lifecycle of NASA CMR granules – i.e. figuring out when a collection’s files actually change, as distinct from CMR metadata churn, so that a downstream mirror (e.g. a COG conversion published to source.coop) knows when it needs to re-derive a file. Originally built against MUR-JPL-L4-GLOB-v4.1; the core is now collection-agnostic, with a companion layer for discovering and sanity-checking collections before you audit them.

Start here

The rest of this README is a compact reference; the vignettes are where the depth and worked examples live.

Installation

From this repo:

# remotes::install_github("hypertidy/cmrlifecycle")

Usage 

library(cmrlifecycle)

# don't have a short_name yet?
cmr_find_collections(keyword = "sea surface temperature", provider = "POCLOUD")

# sanity check before auditing: is it alive, and what does it cover?
cmr_collection_liveness("MUR-JPL-L4-GLOB-v4.1")
cmr_find_collections(short_name = "MUR-JPL-L4-GLOB-v4.1")

# audit the granule-level revision/production lifecycle
rev <- cmr_audit_revisions("MUR-JPL-L4-GLOB-v4.1", from = as.Date("2024-01-01"))
plot_cmr_lifecycle(rev)

See vignette("cmr-quickstart") for the full walkthrough, including opaque-GranuleUR collections (e.g. LAADS swath products), spatial subsetting for high-volume swaths (bounding_box/polygon), and cloud/S3 access.

What cmr_audit_revisions() gives you

One row per granule:

column meaning
granule_ur, concept_id CMR identifiers
data_url first GET DATA (https) download URL from UMM RelatedUrls
s3_url first GET DATA VIA DIRECT ACCESS (s3://) URL – NA if the collection doesn’t publish direct S3 access
n_rev CMR meta revision-id – how many times the metadata record was touched (often bulk churn, not a data-change signal)
insert_date / update_date UMM ProviderDates Insert/Update
rev_date CMR meta revision-date (latest revision)
production DataGranule.ProductionDateTime – when the science file itself was generated. This is the churn-immune signal.
temporal_start UMM TemporalExtent Beginning/SingleDateTime – the granule’s acquisition start
*_lag each date above minus gran_date (days)

gran_date is parsed from a leading YYYYMMDD in granule_ur by default (GHRSST-style naming); pass gran_date_fun for other conventions – see vignette("cmr-quickstart")’s “when your collection doesn’t play by MUR’s rules” section for the opaque- GranuleUR and swath-volume cases this covers.

Key finding this was built to support

n_rev is metadata churn, not a change-of-data-file signal; production is the reliable one. The staleness rule for a mirror is:

stale <- mirror_mtime < as.Date(production)

joined against e.g. rclone lsjson mtimes of the mirror; nc_bytes is a free secondary check (a regenerated file rarely keeps its exact size).

Notes / limits

  • all_revisions=true is not supported for granule searches (CMR 400s) – only the first Insert, latest Update, and a revision count are recoverable, not intermediate revision timestamps.
  • data_url/s3_url pick the first matching UMM RelatedUrls entry; inspect RelatedUrls directly if a collection publishes multiple mirrors and you need a different one.
  • s3_url is only cheap/direct from compute in the same AWS region as the bucket, and always requires an Earthdata Login credential exchange – check cmr_find_collections()’s region/ s3_credentials_endpoint first. See vignette("cmr-landscape") for why this matters.
  • ends_at_present/temporal_end (declared collection-level metadata) vary a lot in reliability by provider – use cmr_collection_liveness() to check real recent granule activity instead when it matters. See vignette("cmr-landscape") for the full investigation (PO.DAAC vs. LAADS at catalog scale).
  • Not every CMR “collection” has granules – cmr_find_collections()’s granule_count flags this directly; see vignette("cmr-landscape") for a real example (an AADC record with a single static download and zero UMM-G granules).

Files

  • R/cmr-lifecycle.R – cmr_find_granules(), cmr_audit_revisions(), plot_cmr_lifecycle()
  • R/cmr-collections.R – cmr_find_collections(), cmr_all_collections(), cmr_collection_liveness()
  • R/globals.R, R/cmrlifecycle-package.R – package plumbing (NSE variable declarations, package-level doc)
  • vignettes/cmr-quickstart.Rmd – live, in-depth single-collection walkthrough
  • vignettes/cmr-landscape.Rmd – narrative write-up of the broader exploration, with baked-in evidence
  • vignettes/cmr-current-snapshot.Rmd – lightweight companion that re-runs a subset of the landscape checks live on every build
  • snapshots/ – dated, rendered copies of cmr-current-snapshot.Rmd, kept as a historical record (not part of the built package)
  • data-raw/cmr-granules.R – the original MUR-specific driver script (not installed as part of the package), kept as a worked example
  • data-raw/cmr_revisions_*.parquet – output of a run of that script