Fetch minimal metadata for every CMR collection matching a search

Pages through cmr_find_collections() with page_num, pausing delay seconds between requests, until every matching collection has been fetched (or max_pages is hit). Useful for a full catalog sweep – e.g. cmr_all_collections() with no filters returns minimal metadata for all ~45,000 CMR collections, in ~23 requests at the default page_size.

Usage

cmr_all_collections(
  keyword = NULL,
  short_name = NULL,
  provider = NULL,
  page_size = 2000,
  delay = 0.3,
  abstract = FALSE,
  max_pages = Inf,
  granule_count = TRUE
)

Arguments

keyword, short_name, provider: As in cmr_find_collections(), all optional here. If both keyword and short_name are NULL (the default), keyword = "" is used, which CMR treats as "match everything" – i.e. the default sweeps the entire CMR catalog.
page_size: Collections per request (CMR max, and the default here, is 2000).
delay: Seconds to pause between requests. Not a documented CMR rate limit – just a politeness default for an unauthenticated, multi-request sweep. Set to 0 to disable.
abstract: Keep the abstract text column. Default FALSE: for a full-catalog sweep the abstracts add real memory/parse cost for a "which collections exist" use case; the network request still fetches them (CMR doesn't support field projection here), this just drops them from the result. Pass TRUE to keep them, as cmr_find_collections() does.
max_pages: Safety cap on the number of requests issued. Inf by default; set lower if you're experimenting with a filter and want to sample rather than fetch everything.
granule_count: As in cmr_find_collections(). Default TRUE, but note this means every page issues two requests (one for collection metadata, one for granule counts) instead of one – for a full-catalog sweep that's ~46 requests rather than ~23. Set FALSE for a faster sweep if you don't need it.

Value

A tibble::tibble(), same columns as cmr_find_collections() (minus abstract unless abstract = TRUE), one row per collection. Reports progress via message() after each page.

Details

CMR's classic page_num/page_size paging is only documented to work up to roughly 1,000,000 records (page_num * page_size); this function stops with a warning if it would exceed that, rather than silently looping. That ceiling is far above the current collection count, so it's a safeguard, not an expected code path.

Examples

# every CMR collection, minimal columns (~23 requests, a few seconds
# of `delay` pauses on top of network time)
# \donttest{
all_colls <- cmr_all_collections()
#> page 1: 2000/45442 collections fetched
#> page 2: 4000/45442 collections fetched
#> page 3: 6000/45442 collections fetched
#> page 4: 8000/45442 collections fetched
#> page 5: 10000/45442 collections fetched
#> page 6: 12000/45442 collections fetched
#> page 7: 14000/45442 collections fetched
#> page 8: 16000/45442 collections fetched
#> page 9: 18000/45442 collections fetched
#> page 10: 20000/45442 collections fetched
#> page 11: 22000/45442 collections fetched
#> page 12: 24000/45442 collections fetched
#> page 13: 26000/45442 collections fetched
#> page 14: 28000/45442 collections fetched
#> page 15: 30000/45442 collections fetched
#> page 16: 32000/45442 collections fetched
#> page 17: 34000/45442 collections fetched
#> page 18: 36000/45442 collections fetched
#> page 19: 38000/45442 collections fetched
#> page 20: 40000/45442 collections fetched
#> page 21: 42000/45442 collections fetched
#> page 22: 44000/45442 collections fetched
#> page 23: 45442/45442 collections fetched

# narrower sweeps are quick
cmr_all_collections(provider = "LAADS")
#> page 1: 188/188 collections fetched
#> # A tibble: 188 × 12
#>    short_name    entry_title concept_id provider version processing_level region
#>    <chr>         <chr>       <chr>      <chr>    <chr>   <chr>            <chr> 
#>  1 VJ102IMG      "VIIRS/JPS… C21050839… LAADS    2.1     1B               us-we…
#>  2 VJ103IMG      "VIIRS/JPS… C21050862… LAADS    2.1     1A               us-we…
#>  3 VNP03IMG      "VIIRS/NPP… C21050921… LAADS    2       1A               us-we…
#>  4 VNP02IMG      "VIIRS/NPP… C21050915… LAADS    2       1B               us-we…
#>  5 VNP02DNB      "VIIRS/NPP… C21050913… LAADS    2       1B               us-we…
#>  6 VJ146A1       "VIIRS/JPS… C33707521… LAADS    2       3                us-we…
#>  7 CLDMSK_L2_VI… "VIIRS/SNP… C38600453… LAADS    2       2                us-we…
#>  8 CLDMSK_L2_VI… "VIIRS/NOA… C38600498… LAADS    2       2                us-we…
#>  9 VNP02MOD      "VNP02MOD … C21050876… LAADS    2       1B               us-we…
#> 10 VJ102MOD      "VIIRS/JPS… C21038117… LAADS    2.1     1B               us-we…
#> # ℹ 178 more rows
#> # ℹ 5 more variables: s3_credentials_endpoint <chr>, temporal_start <chr>,
#> #   temporal_end <chr>, ends_at_present <lgl>, granule_count <int>
# }