Knowledge Base

Programmatically access data using spatial and temporal filters


This article provides a step-by-step getting started guide to utilizing an Application Programming Interface, or API, for programmatic access to data from the NSIDC Distributed Active Archive Center (DAAC) based on spatial and temporal filters. Programmatic access is provided via an HTTPS URL containing a series of key-value-pairs (KVPs), and data can be returned as either a single file or a multi-file zip. This functionality is currently being enhanced to support Web Coverage Service (WCS) compatibility. 

This document provides a step-by-step guide for accessing data via this API. In addition to this guide, comprehensive documentation provided by the API developers, as well as Frequently Asked Questions, can be found on NASA Earthdata. Please contact NSIDC User Services with any questions or feedback on this service: nsidc@nsidc.org.

API data access is available with the following NSIDC DAAC collections:

Aquarius: Aquarius radiometer and scatterometer soil moisture and polar-gridded data
ASO: Airborne Snow Observatory spectrometer and lidar derived snow data
High Mountain Asia: satellite, model, and in situ derived snow, glacier, and permafrost data
IceBridge: data from Operation IceBridge aircraft missions
MEaSUREs: cryospheric Earth System Data Records from the NASA MEaSUREs program
Nimbus Data Rescue: visible and infrared imagery from the 1960s and 1970s
SnowEX: snow data from airborne multi-sensor and in situ measurements

Step 1: Create Token

A token is needed in order to access data using your Earthdata Login credentials. The token is valid for 30 days. See the following Earthdata Wiki article for more information on this step, including troubleshooting information and an alternative file input option: Creating a Token Common.

curl example:

Make sure to replace the following with your personal credentials in the curl command below:

earthdata_login_user_name
earthdata_login_password
your_origin_ip_address

curl -X POST --header "Content-Type: application/xml" -d "<token><username>earthdata_login_user_name</username><password>earthdata_login_password</password><client_id>NSIDC_client_id</client_id><user_ip_address>your_origin_ip_address</user_ip_address> </token>" https://cmr.earthdata.nasa.gov/legacy-services/rest/tokens

You should receive the token ID in the response. The value in the <id> tag is the token you will use in Step 3. For example:

<token>
<id>75E5CEBE-6BBB-2FB5-A613-0368A361D0B6</id>
<username>earthdata_login_user_name</username>
<client_id>NSIDC_client_id</client_id>
<user_ip_address>your_origin_ip_address</user_ip_address>
</token>

Step 2: Create the API endpoint. This includes the data host, key-value-pairs (KVPs) used to search the Common Metadata Repository (CMR) for requested files, and KVPs to access Earthdata login information via your token from Step 1.

The programmatic access structure is as follows:

https://n5eil02u.ecs.nsidc.org/egi/request?<kvp operands separated with &>

Request example: MEaSUREs Greenland Ice Velocity: Selected Glacier Site Velocity Maps from InSAR, Version 1, collected from 15 August 2017 to 1 September 2017 over the western Greenland coast, specifying no customization service (subsetting, reformatting, reprojection) agent used and no metadata requested:

https://n5eil02u.ecs.nsidc.org/egi/request?short_name=NSIDC-0481&version=1&time=2017-08-15,2017-09-01&bounding_box=-52.5,68.5,-47.5,69.5&agent=NO&INCLUDE_META=N&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6

The programmatic access structure using curl is as follows:

curl <options> <endpoint>/request?<kvp operands separated with &>

Request example using curl commands. In this example, the --dump-header option is used and the response-header.txt returns the HTTP response status code along with file name and/or debugging info. Nine files are returned in a single .zip file to the current working directory.

curl -O -J --dump-header response-header.txt "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=NSIDC-0481&version=1&time=2017-08-15,2017-09-01&bounding_box=-52.5,68.5,-47.5,69.5&agent=NO&INCLUDE_META=N&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6"

The following curl options may also be useful for programmatic access.

-v       verbose, prints internal details of performing the curl request, used for debugging
  -s       silent, suppresses all unnecessary information output, used when not debugging
  -L       follow the referral to a new location, automatically handles the Code 303 redirect 
  -O       save the result in a local file
  -J       name the saved file according to the returned header, used with -O
  -w       write specified info to stdout, used to capture return codes, file names and urls
  -i       show returned headers, useful for debugging
  --dump-header       include the HTTP header in the output, used to get the returned file name

The following table lists important KVPs associated with the Common Metadata Repository (CMR) file search. Please see the CMR Search API Documentation page for additional information on CMR parameters.

Key-value-pair (KVP) operand

Description

Example

short_name

Required field. Specifies the short name ID of the
data set requested.

Can be used multiple times to return files from multiple
data sets.

short_name=ILATM1B

agent

Required field. Specifies that no service agent is used;
native data are returned. The agent must be set to NO.

agent=NO

token Required field. Specifies the token needed to access
data using Earthdata Login credentials.

token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6

email Required field. Specifies whether or not to receive an email with status or error information. 

Request email associated with Earthdata Login account:
email=yes
OR
email=true

Request specified email:
email=name@domain.com

version

Specifies the data set version number. If not specified,
all available versions will be returned. Please note that
this can either be a 3-digit or 1-digit number.
To determine the correct verion number, query CMR
collection metadata.
For example:

curl "https://cmr.earthdata.nasa.gov/search/
collections.json?short_name=ILATM1B&pretty=true"
| grep version_id

version=2

time

Specifies a temporal range filter in start_time,end_time
format. Valid options include:

time=yyyy-MM-dd,yyyy-MM-dd
time=yyyy-MM-ddTHH:mm:ssZ,yyyy-MM-ddTHH:mm:ssZ
time=yyyy-MM-ddTHH:mm:ss,yyyy-MM-ddTHH:mm:ss

If the first option is used (no time specified), the day
defaults to T00:00:00. This parameter is the WCS
compatible equivalent of the CMR 'temporal'
parameter.

time=
2018-06-06,2018-06-07

sort_key

Specifies the sort order of returned files from the CMR
search. If a temporal range search is performed, the
search results will be sorted by temporal overlap
percentage over all ranges provided by default.
Otherwise, results are sorted by ascending entry title by
default. See Sorting Collection Results in the CMR
documentation for more info.

Sort files by descending start date (most recent data first):

sort_key=-start_date

page_num

This is a CMR parameter to select the page of results to
be processed. See Guide to large file requests below
for more guidance on this parameter.

page_num=2

bounding_box

Specifies a search filter to find files with a spatial extent
that overlaps this bounding box, specified in decimal
degrees of latitude and longitude in WSEN format:

bounding_box=lower_left_long,lower_left_lat,
upper_right_long,upper_right_lat

bounding_box=
-109.2348,35.9385,-105.0392,38.1285

INCLUDE_META Specifies whether or not to include the associated XML
metadata. If not specified, metadata are returned by default. 
INCLUDE_META=N

Guide to large file requests 

Request example: High Mountain Asia ASCAT Freeze/Thaw/Melt Status, Version 1, collected from 2017-01-01 to 2018-01-01 (285 files requested plus associated metadata), with email requested:

curl -O -J --dump-header response-header.txt "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=HMA_FreezeThawMelt_ASCAT&version=001&time=2017-01-01,2018-01-01&agent=NO&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=true"

In this example, only 10 of the 285 files requested will be returned. This is due to the default page size of the CMR search results, which is set to 10 results. In order to receive your entire order in the case of 11 or more requested files, the page_num KVP is required to request successive CMR page results. The following set of bash commands loops through the page numbers until all files are returned from the example above:

i=1
while [ $? -eq 0 ]
do
    curl -O -J -L --dump-header response-header.txt "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=HMA_FreezeThawMelt_ASCAT&version=1&time=2017-01-01,2018-01-01&agent=NO&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=true&page_num="$i
    i=$[$i+1]
    grep -E 'Content-Disposition: attachment; filename=".*zip"' response-header.txt
done

This loop will return multi-file zips for each page of CMR results (in this case, 29 zips are returned totaling 285 output files given the filters specified). 

Last updated: September 2019