Beyond One Collection: Lessons from Auditing the CMR Provider Landscape

This article is a written record of a real exploration session with cmrlifecycle, kept as a permanent guide rather than left to scatter across commit messages and chat logs. Every number below was produced by actually running the code shown, against the live CMR API, on 2026-08-25. The code is shown for reproducibility but is not re-executed when this vignette builds – CMR’s catalog changes over time, and a several-hundred-request live sweep isn’t something to repeat on every R CMD check. For a lightweight version of these same questions re-run fresh, see the companion article vignette("cmr-current-snapshot").

If you arrived here only caring about one collection: cmr_find_granules() and cmr_audit_revisions() are the two functions you need, and the README covers them. Keep reading if you’re curious why a granule-lifecycle tool grew a whole collection-discovery layer, and what that layer found.

Summary

• This package started as a MUR-only script and was generalized into a granule-lifecycle auditor (cmr_find_granules(), cmr_audit_revisions()) plus a collection-discovery layer (cmr_find_collections(), cmr_all_collections(), cmr_collection_liveness()) after the same questions kept coming up for other collections.
• A routine liveness check on MUR-JPL-L4-GLOB-v4.1 surfaced a real, silent extraction bug. temporal_start returned the same date for every granule in a chunk whenever the chunk’s UMM-G shape was uniform – the common case. Root cause: jsonlite flattens a single nested JSON object (not an array) into one data.frame when every row matches, instead of keeping a genuine per-granule list; the old code iterated that data.frame’s columns, not its rows. This had also quietly broken this package’s own MOD021KM example. Fixed, with regression tests covering both jsonlite shapes.
• Every other extraction path was individually audited against real, uniformly-shaped CMR responses and confirmed safe – n_rev, insert_date/update_date, rev_date, production, data_url, s3_url, nc_bytes all come from JSON arrays or plain scalar fields, and jsonlite never collapses those the way it collapsed TemporalExtent.
• Not every CMR “collection” has granules. Some records (e.g. an AADC record, AAS_4088_Adelie_occupancy_Mawson_Taylor) register a single static download at the collection level, with zero UMM-G granules underneath. granule_count (from a second, lighter CMR endpoint) flags this directly.
• S3 access has a region gotcha. s3_urls are only cheaply/ directly readable from compute in the same AWS region as the bucket (PO.DAAC is us-west-2), and require an Earthdata Login credential exchange regardless of region. Both are surfaced by cmr_find_collections().
• ends_at_present reliability depends on the provider, not on CMR as a whole. PO.DAAC leaves it FALSE/unset on ~95% of its ~982 collections – including collections that are demonstrably still producing granules daily. LAADS sets it reliably on ~71% of its ~184 collections. Treat the flag as provider-dependent metadata quality, not a CMR-wide convention, and use cmr_collection_liveness() to check the real thing when it matters.

The landscape: collections, granules, and this package’s tools

CMR (NASA’s Common Metadata Repository) indexes two related but distinct kinds of record:

• a collection is a dataset/product entry – one row per short_name, e.g. MUR-JPL-L4-GLOB-v4.1 – carrying metadata like title, abstract, declared temporal coverage, and (for cloud-hosted data) AWS region and S3 credentials endpoint.
• a granule is one file within a collection – one row per granule_ur, e.g. one day’s MUR SST grid.

cmrlifecycle’s functions split cleanly along that line:

question	function
What collections exist matching a keyword/provider?	cmr_find_collections()
Every collection matching a filter (or the whole catalog)?	cmr_all_collections()
One page of granules for a date window (+ optional space)?	cmr_find_granules()
The full revision/production lifecycle of a collection’s granules?	cmr_audit_revisions()
Is this collection still actually producing granules?	cmr_collection_liveness()
Standard lifecycle diagnostic plots	plot_cmr_lifecycle()

The natural workflow is left-to-right in that table: discover a collection, decide whether you trust its stated liveness/temporal metadata, then audit its granules. Below is the narrative of how each row of that table came to exist.

Where it started: auditing one collection

The original question was narrow and practical: for MUR-JPL-L4-GLOB-v4.1 (a daily global sea-surface-temperature grid), when does a granule’s underlying science file actually change, as opposed to its CMR metadata being touched without the data changing? cmr_audit_revisions() answers this by pulling revision-id (a metadata-churn counter), ProviderDates (Insert/Update), and DataGranule.ProductionDateTime (the churn-immune “science file was regenerated” signal) for every granule in a date range.

The first two real-world findings were about trusting the wrong column:

rev <- cmr_audit_revisions("MUR-JPL-L4-GLOB-v4.1", from = as.Date("2024-01-01"))
range(rev$n_rev)
#> [1]  4 23

n_rev (CMR’s revision-id) ranged from 4 to 23 – but this turned out to be almost entirely metadata churn (bulk provider events), not evidence the underlying file changed 23 times. production (DataGranule.ProductionDateTime) is the reliable signal.

A second, more subtle bug: nc_bytes (file size, used as a secondary staleness check) initially picked ArchiveAndDistributionInformation[1] as “the” file, assuming checksum sidecars (.md5) always came last. They don’t. Re-running the full MUR audit after fixing the extraction to exclude sidecar extensions and take the largest remaining entry:

# before the fix: ArchiveAndDistributionInformation[1]
# after the fix: exclude .md5/.cmr/.xml, take the largest remaining entry
sum(nc_bytes_old != nc_bytes_fixed) / length(nc_bytes_fixed)
#> [1] 0.4548 # 440 of 967 rows changed from ~95 bytes to the real ~450-780 MB size

Lesson for a new user: don’t assume the first element of a UMM-G array field is the one you want. CMR’s arrays aren’t guaranteed ordered by role.

Generalizing: finding out what’s out there

Once the granule-lifecycle tool was collection-agnostic, the next question was discovery: what collections exist, and how do I find one without already knowing its short_name? cmr_find_collections() wraps CMR’s collections.umm_json?keyword=... search:

cmr_find_collections(keyword = "sea surface temperature", provider = "POCLOUD")
#> 100 of 499 total matching collections returned -- raise `page_size` ...

An empty keyword turns out to match everything:

cmr_find_collections(keyword = "")
#> 100 of 45431 total matching collections returned -- raise `page_size` ...

…which is a fine trick, but not something to lean on by hand for a 45,000-row catalog – cmr_all_collections() was built to page through that automatically (with a politeness delay between requests), landing at 45,431 collections in ~23 requests.

Not every collection has granules

Browsing that catalog surfaced AAS_4088_Adelie_occupancy_Mawson_Taylor, an Australian Antarctic Data Centre (AADC) record. cmr_find_granules() returned nothing for it:

cmr_find_granules("AAS_4088_Adelie_occupancy_Mawson_Taylor",
                   as.Date("2015-01-01"), as.Date("2015-12-31"))
#> NULL

Not a bug – this collection has zero UMM-G granule records at all. It registers a single static download (http://data.aad.gov.au/eds/4405/download) directly on the collection metadata’s RelatedUrls, a pattern common for field-survey/social-science-style datasets imported into CMR as one file rather than a time series. This motivated adding granule_count to cmr_find_collections(), via a second, lighter request to collections.json?include_granule_counts=true (the only CMR endpoint that reports it):

cmr_find_collections(short_name = "AAS_4088_Adelie_occupancy_Mawson_Taylor")[
  , c("short_name", "granule_count", "temporal_start", "temporal_end")]
#> # A tibble: 1 x 4
#>   short_name                              granule_count temporal_start  temporal_end
#>   AAS_4088_Adelie_occupancy_Mawson_Taylor              0 2015-02-15T00:… 2015-02-15T…

Lesson: granule_count == 0 is the direct signal for a granule-less collection; don’t infer it from an empty cmr_find_granules() call when you can ask CMR outright. For these collections, temporal_start/temporal_end from UMM-C TemporalExtents is the only date range CMR has at all.

The S3 region gotcha

VIIRSN_L3m_CHL (ocean chlorophyll) publishes both an https mirror and a direct s3:// link:

x <- cmr_find_granules("VIIRSN_L3m_CHL", Sys.Date() - 100, Sys.Date())
x$s3_url[1]
#> [1] "s3://ob-cumulus-prod-public/SNPP_VIIRS...9km.nc"

The catch: that s3:// URL is only cheap and direct from compute running in the same AWS region as the bucket. cmr_find_collections() surfaces that region (and the credentials endpoint you need regardless of region, since s3:// reads aren’t public):

cmr_find_collections(short_name = "VIIRSN_L3m_CHL")[
  , c("short_name", "region", "s3_credentials_endpoint")]
#> # A tibble: 1 x 3
#>   short_name      region    s3_credentials_endpoint
#>   VIIRSN_L3m_CHL  us-west-2 https://obdaac-tea.earthdatacloud.nasa.gov/s3credentials

Lesson: check region before committing compute placement. From outside the matching region, s3_url is no faster than data_url and may incur egress charges or be blocked by bucket policy.

A silent correctness bug hiding in plain sight

Cross-checking MUR-JPL-L4-GLOB-v4.1’s declared collection-level temporal_start against a real granule sweep should have been a formality – and it was, for the first granule. But every granule in the window showed the same date:

mur_early <- cmr_find_granules("MUR-JPL-L4-GLOB-v4.1",
                                as.Date("2002-05-25"), as.Date("2002-06-10"))
mur_early[order(mur_early$temporal_start), c("granule_ur", "temporal_start")]
#> # before the fix -- every row identical, which is wrong:
#>  1 20020601090000-...    2002-05-31T21:00:00.000Z
#>  2 20020602090000-...    2002-05-31T21:00:00.000Z
#>  ...
#> 11 20020611090000-...    2002-05-31T21:00:00.000Z

The cause: UMM-G’s TemporalExtent is a single nested JSON object (not an array). When every granule in a chunk shares the same shape – the common case – jsonlite::fromJSON(..., simplifyDataFrame = TRUE) flattens it into one data.frame (one row per granule) rather than a genuine per-granule list. The old extraction code did vapply(te, .get_temporal_start, ""), which iterates a data.frame’s columns, not its rows – with only one column (RangeDateTime), that returned a length-1 result recycled across all 11 rows: the first granule’s date, silently repeated.

This wasn’t a MUR-only quirk – the exact same collapse hit the package’s own MOD021KM spatial-filtering example. MOD021KM (a MODIS swath product) can produce hundreds of granules a day worldwide, so cmr_find_granules()/cmr_audit_revisions() take an optional bounding_box/polygon filter to subset to a region of interest – for 2026-08-01, a bounding box over eastern Australia cut 552 granules down to 8:

mod <- cmr_find_granules("MOD021KM", as.Date("2026-08-01"), as.Date("2026-08-01"),
                          bounding_box = c(140, -45, 155, -35))
mod[, c("granule_ur", "temporal_start")]
#> # before the fix -- all 8 granules show the identical timestamp:
#>   granule_ur       temporal_start
#>   LAADS:9975746428 2026-08-01T11:15:00.000Z
#>   LAADS:9980651751 2026-08-01T11:15:00.000Z
#>   ... (6 more, all identical)

After the fix (.extract_temporal_starts(), which detects and handles both the flattened-data.frame shape and the genuine list-column shape):

mur_early2 <- cmr_find_granules("MUR-JPL-L4-GLOB-v4.1",
                                 as.Date("2002-05-25"), as.Date("2002-06-10"))
mur_early2[order(mur_early2$temporal_start), c("granule_ur", "temporal_start")]
#> # after the fix -- correctly increasing:
#>  1 20020601090000-...    2002-05-31T21:00:00.000Z
#>  2 20020602090000-...    2002-06-01T21:00:00.000Z
#>  3 20020603090000-...    2002-06-02T21:00:00.000Z
#>  ...
#> 11 20020611090000-...    2002-06-10T21:00:00.000Z

mod2 <- cmr_find_granules("MOD021KM", as.Date("2026-08-01"), as.Date("2026-08-01"),
                           bounding_box = c(140, -45, 155, -35))
mod2[, c("granule_ur", "temporal_start")]
#> # after the fix -- 4 distinct swath-pass timestamps across the 8 granules:
#>   LAADS:9975746428 2026-08-01T11:15:00.000Z
#>   LAADS:9980651751 2026-08-01T11:15:00.000Z
#>   LAADS:9975763948 2026-08-01T12:50:00.000Z
#>   LAADS:9980653369 2026-08-01T12:50:00.000Z
#>   LAADS:9975774200 2026-08-01T22:15:00.000Z
#>   LAADS:9980668284 2026-08-01T22:15:00.000Z
#>   LAADS:9975768761 2026-08-01T22:20:00.000Z
#>   LAADS:9980660739 2026-08-01T22:20:00.000Z

This didn’t just mean missing dates – it meant wrong ones, silently. The package’s own “0 NAs” check for the MOD021KM example had passed throughout, because the bug produces a value, just the wrong one, for every row after the first.

Once found, the natural next question was: is anything else in the pipeline vulnerable to the same collapse? Every other extraction path was checked against a real, fully-uniform MUR chunk (every granule with exactly 2 ProviderDates entries, 6 RelatedUrls entries, 2 ArchiveAndDistributionInformation entries):

str(j$items$umm$ProviderDates)
#> List of 6            <- genuine per-granule list, even though every
#>  $ :'data.frame': 2 obs. of 2 variables    element has the same shape
#>  $ :'data.frame': 2 obs. of 2 variables
#>  ...

ProviderDates, RelatedUrls, and DataGranule$ArchiveAndDistributionInformation are JSON arrays in UMM-G, and jsonlite reliably keeps a JSON array as a genuine per-granule list-column regardless of shape uniformity – only a single (non-array) nested object like TemporalExtent collapses. n_rev/rev_date (meta fields) and production (DataGranule$ProductionDateTime) are plain scalar columns on always-proper per-granule data.frames, never list-columns at all. TemporalExtent was the only field in the pipeline actually at risk, confirmed rather than assumed.

Lesson: a “0 NAs” or “no error” check is not the same as a correctness check. Cross-checking a derived value against an independent source (here: the collection’s own declared temporal_start) is what actually caught this.

Is this collection still alive? The ends_at_present rabbit hole

UMM-C’s TemporalExtents carries an EndsAtPresentFlag – in principle, exactly the “is this collection still growing” signal a user would want. In practice:

shorts <- c("MUR-JPL-L4-GLOB-v4.1", "VIIRSN_L3m_CHL", "AVHRR_OI-NCEI-L4-GLOB-v2.1",
            "CMC0.1deg-CMC-L4-GLOB-v3.0", "MODIS_T-JPL-L2P-v2019.0",
            "H09-AHI-L3C-ACSPO-v2.90")
# ends_at_present, declared:
#>   MUR-JPL-L4-GLOB-v4.1        FALSE
#>   VIIRSN_L3m_CHL               TRUE
#>   AVHRR_OI-NCEI-L4-GLOB-v2.1  FALSE
#>   CMC0.1deg-CMC-L4-GLOB-v3.0  FALSE
#>   MODIS_T-JPL-L2P-v2019.0     FALSE
#>   H09-AHI-L3C-ACSPO-v2.90     FALSE

# granules in the last 14 days, actual:
#>   MUR-JPL-L4-GLOB-v4.1          14
#>   AVHRR_OI-NCEI-L4-GLOB-v2.1    15
#>   CMC0.1deg-CMC-L4-GLOB-v3.0    15
#>   MODIS_T-JPL-L2P-v2019.0     2000  # hit the per-request page cap
#>   H09-AHI-L3C-ACSPO-v2.90      355

Only 1 of 6 correctly declared TRUE. The other 5 all declared FALSE (with temporal_end = NA) despite every one having real, recent granules – one hitting the 2000-per-request page cap outright. Six collections isn’t a sample size to generalize from, though, so the same check was repeated at the scale of an entire provider’s catalog, for two providers with very different pipelines:

poc <- cmr_all_collections(provider = "POCLOUD")   # 982 collections, 2 requests
table(poc$ends_at_present)
#>  FALSE  TRUE
#>    938    44

laads <- cmr_all_collections(provider = "LAADS")   # 184 collections, 2 requests
table(laads$ends_at_present)
#>  FALSE  TRUE
#>     54   130

provider	collections	ends_at_present = TRUE	ambiguous*
POCLOUD	982	4.5%	33.7%
LAADS	184	70.7%	2.2%

*“ambiguous” = FALSE, no declared end date, but granule_count > 0.

A random sample of 25 of PO.DAAC’s 331 “ambiguous” collections found 15 (60%) had granules landing within the last 60 days – genuinely active, just never flagged. LAADS’s own claims held up under the same scrutiny: a sample of 15 TRUE-declared collections was 14/15 (93%) actually active in the last 60 days; a sample of 15 “declared closed” collections was 15/15 (100%) genuinely silent past their declared end date; even LAADS’s tiny 4-collection ambiguous bucket showed the same failure mode at a smaller scale (3/4 active despite the unset flag).

Lesson: ends_at_present reliability is a property of which DAAC maintains the collection’s metadata pipeline, not a CMR-wide guarantee. PO.DAAC (ocean/GHRSST products in this sample) essentially doesn’t populate it; LAADS (MODIS/VIIRS atmosphere/land products) does, accurately. Don’t assume either direction applies to a provider you haven’t checked.

This is what motivated cmr_collection_liveness(): rather than trust a flag whose reliability varies by provider, check real recent granule activity directly.

cmr_collection_liveness("MUR-JPL-L4-GLOB-v4.1")
#> # A tibble: 1 x 5
#>   short_name           provider lookback_days recent_granule_count is_live
#>   MUR-JPL-L4-GLOB-v4.1 NA                   60                   60 TRUE

# pipe-friendly, straight from a filtered collection search
cmr_all_collections(provider = "POCLOUD") |>
  dplyr::filter(!ends_at_present, is.na(temporal_end), granule_count > 0) |>
  cmr_collection_liveness()

Even building this small helper turned up one more instance of the same general class of bug this whole session kept surfacing – silent wrong output rather than a loud error. An early version passed a NA provider straight into the request URL, producing a literal &provider=NA query string that CMR quietly interpreted as “no such provider” and returned zero granules for a collection that plainly has them (60 granules in 60 days for MUR). Fixed by treating a NA provider as “no filter” before building the request.

Lesson, one more time: in a codebase talking to an API this large and this heterogeneous, the failure mode to design tests against isn’t “the request errors” – it’s “the request succeeds and returns a plausible-looking, wrong answer.” Every bug in this narrative was exactly that shape.

A quick-start for a new user

If none of the above is what brought you here, and you just want one collection’s lifecycle:

library(cmrlifecycle)

# find a short_name if you don't already have one
cmr_find_collections(keyword = "your topic here")

# check it's actually alive and see its declared coverage
cmr_collection_liveness("your-short-name")
cmr_find_collections(short_name = "your-short-name")[
  , c("temporal_start", "temporal_end", "ends_at_present", "granule_count")]

# audit its granule-level revision/production lifecycle
rev <- cmr_audit_revisions("your-short-name", from = as.Date("2024-01-01"))
plot_cmr_lifecycle(rev)

See the package README for the full column dictionary, spatial-filtering (bounding_box/ polygon) options for swath products, and S3-access caveats.