Knowledge Base

How do I programmatically access data using spatial and temporal filters?


The Common Metadata Repository (CMR) is a high-performance metadata system that provides search capabilities for data at NSIDC. A synchronous REST interface was developed which utilizes the CMR API, allowing you to programmatically access data from the NSIDC Distributed Active Archive Center (DAAC) based on spatial and temporal filters. The resulting output data are returned as either a single file or a multi-file zip. This functionality is currently being enhanced to support Web Coverage Service (WCS) compatibility. 

This document provides a step-by-step guide for accessing data via this API. In addition to this guide, comprehensive documentation provided by the API developers, as well as Frequently Asked Questions, can be found on the Earthdata Developer Portal. Please contact NSIDC User Services with any questions or feedback on this service: nsidc@nsidc.org.

API data access is available with the following NSIDC DAAC collections:

Aquarius: Aquarius radiometer and scatterometer soil moisture and polar-gridded data
ASO: Airborne Snow Observatory spectrometer and lidar data
High Mountain Asia: satellite, model, and in situ derived snow and glacier data
IceBridge: data from Operation IceBridge aircraft missions
MEaSUREs: cryospheric Earth System Data Records from the NASA MEaSUREs program
Nimbus Data Rescue: visible and infrared imagery from the 1960s and 1970s
SMMR-SSM/I-SSMIS: snow and ice data from passive microwave sensors
SnowEX: snow data from airborne multi-sensor and in situ measurements

Step 1: Create Token

A token is needed in order to access data using your Earthdata Login credentials. The token is valid for 30 days. See the following Earthdata Wiki article for more information on this step, including troubleshooting information and an alternative file input option: Creating a Token Common.

curl example (replace all items in bold with your personal credentials):

curl -X POST --header "Content-Type: application/xml" -d '<token><username>earthdata_login_user_name</username><password>earthdata_login_password</password><client_id>NSIDC_client_id</client_id><user_ip_address>your_origin_ip_address</user_ip_address> </token>' https://api.echo.nasa.gov/echo-rest/tokens

You should receive the token ID in the response. The value in the ID tag is the token you will use in Step 2. For example:

<token>
<id>75E5CEBE-6BBB-2FB5-A613-0368A361D0B6</id>
<username>earthdata_login_user_name</username>
<client_id>NSIDC_client_id</client_id>
<user_ip_address>your_origin_ip_address</user_ip_address>
</token>

Step 2: Create the API endpoint. This includes the data host, key-value-pairs (KVPs) used to search the Common Metadata Repository (CMR) for requested files, and KVPs to access Earthdata login information via your token from Step 1.

The programmatic access structure is as follows:

https://n5eil02u.ecs.nsidc.org/egi/request?search_kvp=search_kvp&token=token&email=email

Request example: MEaSUREs Greenland Ice Velocity: Selected Glacier Site Velocity Maps from InSAR, Version 1, collected from 15 August 2017 to 1 September 2017 over the western Greenland coast: 

https://n5eil02u.ecs.nsidc.org/egi/request?short_name=NSIDC-0481&version=1&time=2017-08-15,2017-09-01&bounding_box=-52.5,68.5,-47.5,69.5&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=name@domain.com

The programmatic access structure using curl is as follows:

curl <options> <endpoint>/request?<parameters separated with &>

Request example using curl commands. In this example, the --dump-header option is used and the response-header.txt returns the HTTP response status code along with file name and/or debugging info. Nine granules are returned in a single .zip file to the current working directory.

curl -O -J --dump-header response-header.txt "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=NSIDC-0481&version=1&time=2017-08-15,2017-09-01&bounding_box=-52.5,68.5,-47.5,69.5&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=name@domain.com"

The following curl options may also be useful for programmatic access.

  -v       verbose, prints internal details of performing the curl request, used for debugging
  -s       silent, suppresses all unnecessary information output, used when not debugging
  -L       follow the referral to a new location, automatically handles the Code 303 redirect 
  -O       save the result in a local file
  -J       name the saved file according to the returned header, used with -O
  -w       write specified info to stdout, used to capture return codes, file names and urls
  -i       show returned headers, useful for debugging
  --dump-header       include the HTTP header in the output, used to get the returned file name

The following table lists important KVPs associated with the Common Metadata Repository (CMR) file search. Please see the CMR Search API Documentation page for additional information on CMR parameters.

Key-value-pair operand

Description

Example

short_name

Required field. Specifies the short name ID of the
data set requested.

Can be used multiple times to return files from multiple
data sets.

short_name=ILATM1B

email

Required field. Specifies an email address to be used
for contact information.

email=name@domain.com

version

Specifies the data set version number. If not specified,
all available versions will be returned. Please note that
this can either be a 3-digit or 1-digit number.
To determine the correct verion number, query CMR
collection metadata.
For example:

curl "https://cmr.earthdata.nasa.gov/search/
collections.json?short_name=ILATM1B&pretty=true"
| grep version_id

version=2

time

Specifies a temporal range filter in start_time,end_time
format. Valid options include:

time=yyyy-MM-dd,yyyy-MM-dd
time=yyyy-MM-ddTHH:mm:ssZ,yyyy-MM-ddTHH:mm:ssZ
time=yyyy-MM-ddTHH:mm:ss,yyyy-MM-ddTHH:mm:ss

If the first option is used (no time specified), the day
defaults to T00:00:00. This parameter is the WCS
compatible equivalent of the CMR 'temporal'
parameter.

time=
2018-06-06,2018-06-07

sort_key

Specifies the sort order of returned files from the CMR
search. If a temporal range search is performed, the
search results will be sorted by temporal overlap
percentage over all ranges provided by default.
Otherwise, results are sorted by ascending entry title by
default. See Sorting Collection Results in the CMR
documentation for more info.

Sort files by descending start date (most recent data first):

sort_key=-start_date

page_num

This is a CMR parameter to select the page of results to
be processed. See Guide to large file requests below
for more guidance on this parameter.

page_num=2

bounding_box

Specifies a search filter to find files with a spatial extent
that overlaps this bounding box, specified in decimal
degrees of latitude and longitude in WSEN format:

bounding_box=lower_left_long,lower_left_lat,
upper_right_long,upper_right_lat

bounding_box=
-109,35,-105,38

Guide to large file requests 

Request example: High Mountain Asia ASCAT Freeze/Thaw/Melt Status, Version 1, collected from 2017-01-01 to 2018-01-01 (285 files requested):

curl -O -J --dump-header response-header.txt "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=HMA_FreezeThawMelt_ASCAT&version=001&time=2017-01-01,2018-01-01&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=name@domain.com"

In this example, only 10 of the 285 files requested will be returned. This is due to the default page size of the CMR search results, which is set to 10 results. In order to receive your entire order in the case of 11 or more requested files, the page_num KVP is required to request successive CMR page results. The following set of bash commands loops through the page numbers until all files are returned from the example above:

i=1
while [ $? -eq 0 ]
do
    curl -O -J -L --dump-header response-header.txt 'https://n5eil02u.ecs.nsidc.org/egi/request?short_name=HMA_FreezeThawMelt_ASCAT&version=1&time=2017-01-01,2018-01-01&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=name@domain.com&page_num='$i
    i=$[$i+1]
    grep -E 'Content-Disposition: attachment; filename=".*zip"' response-header.txt
done

This loop will return multi-file zips for each page of CMR results (in this case, 29 zipped files are returned totaling 285 file outputs). 

Last updated: 16 November 2018

86,5 Bot