Knowledge Base

How do I programmatically request data services such as subsetting, reformatting, and reprojection using an API?

This article provides a step-by-step getting started guide to utilizing the Common Metadata Repository (CMR) API. CMR is a metadata system that provides search capabilities for data at NSIDC. A synchronous REST interface utilizes the CMR API, enabling programmatic access to data from the NSIDC Distributed Active Archive Center (DAAC) based on spatial and temporal filters, as well as subsetting, reformatting, and reprojection services available through NASA Earthdata Search. Programmatic access is provided via an HTTPS URL containing a series of key-value-pairs (KVPs), and data can be returned as either a single file or a multi-file zip. This functionality is currently being enhanced to support Web Coverage Service (WCS) compatibility. Comprehensive documentation provided by the API developers, as well as Frequently Asked Questions, can be found on the Earthdata Developer Portal.

For questions about accessing NSIDC data using this API, please contact NSIDC User Services: nsidc@nsidc.org

Step 1: Determine the services enabled for your data set(s) of interest

The subsetting, reformatting, and reprojection services provided by NSIDC apply to SMAP, MODIS, ICESat-2, ICESat/GLAS, AMSR-E, and VIIRS data, and are unique to each of these data sets. 

The following tables list an overview of services provided for each collection:

  1. SMAP
  2. MODIS
  3. ICESat-2
  4. ICESat/GLAS
  5. AMSR-E
  6. VIIRS


Step 2: Create Token

A token is needed in order to access data using your Earthdata Login credentials. The token is valid for 30 days. See the following Earthdata Wiki article for more information on this step, including troubleshooting information and an alternative file input option: Creating a Token Common.

curl example (replace all items in bold with your personal credentials):

curl -X POST --header "Content-Type: application/xml" -d "<token><username>earthdata_login_user_name</username><password>earthdata_login_password</password><client_id>NSIDC_client_id</client_id><user_ip_address>your_origin_ip_address</user_ip_address> </token>" https://cmr.earthdata.nasa.gov/legacy-services/rest/tokens

You should receive the token ID in the response. The value in the <id> tag is the token you will use in Step 3. For example:

<token>
<id>75E5CEBE-6BBB-2FB5-A613-0368A361D0B6</id>
<username>earthdata_login_user_name</username>
<client_id>NSIDC_client_id</client_id>
<user_ip_address>your_origin_ip_address</user_ip_address>
</token>


Step 3: Create the API endpoint.

This includes the data host, KVPs used to search the Common Metadata Repository (CMR) for requested files, KVPs used to request data services, and KVPs to access Earthdata login information via your token and email from Step 2.

The programmatic access structure is as follows:

https://n5eil02u.ecs.nsidc.org/egi/request?<kvp operands separated with &>


Request example 1: SMAP L3 Radiometer Global Daily 36 km EASE-Grid Soil Moisture, Version 5, GeoTIFF reformatting, spatial subsetting*, parameter subsetting, and Geographic reprojection for all data collected 2008-06-06 to 2018-06-07 over Colorado:
 

https://n5eil02u.ecs.nsidc.org/egi/request?short_name=SPL3SMP&version=005&format=GeoTIFF&time=2018-06-06,2018-06-07&bounding_box=-109,37,-102,41&bbox=-109,37,-102,41&Coverage=/Soil_Moisture_Retrieval_Data_AM/
soil_moisture&projection=Geographic&email=yes&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6

*For all bounding box spatial subsetting requests, you must use both the bounding_box search filter KVP and the bbox subsetting KVP, and these coordinates must be identical in order to avoid errors.



Request example 2: ATLAS/ICESat-2 L3A Land Ice Height, Version 1, with variable subsetting for all data over the Barnes Ice Cap using an uploaded shapefile* of the ice cap boundary:
 

curl -O -J -F "shapefile=@barnes_shapefile.zip" "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=ATL06&version=001&polygon=-74.886,69.408,-71.681,69.408,-71.681,70.62,-74.886,70.62,
-74.886,69.408&coverage=/gt1l/land_ice_segments/h_li,/gtl1/land_ice_segments/latitude,
/gtl1/land_ice_segments/longitude,/gtlr/land_ice_segments/h_li,/gtlr/land_ice_segments/latitude,
/gtlr/land_ice_segments/longitude,/gt2l/land_ice_segments/h_li,/gt21/land_ice_segments/latitude,
/gt21/land_ice_segments/longitude,/gt2r/land_ice_segments/h_li,/gt2r/land_ice_segments/latitude,
/gt2r/land_ice_segments/longitude,/gt31/land_ice_segments/h_li,/gt31/land_ice_segments/latitude,
/gt31/land_ice_segments/longitude,/gt3r/land_ice_segments/h_li,/gt3r/land_ice_segments/latitude,
/gt3r/land_ice_segments/longitude,/quality_assessment&email=yes&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6"

*This example includes the curl POST command needed to upload a Shapefile or KML to the subsetter. This functionality is currently only available for ICESat-2 data. The polygon search filter KVP is used in addition to the file upload used for subetting. See below for more guidance on uploading a file to be used for polygon subsetting. 



Request example 3: MODIS/Terra Snow Cover Daily L3 Global 500m SIN Grid, Version 6, collected from 2018-05-31 17:03:36 to 2018-06-01 06:47:53 over Iceland, without customization services requested (native data)*:
 

https://n5eil02u.ecs.nsidc.org/egi/request?short_name=MOD10A1&version=6&time=2018-05-31T17:03:36,2018-06-01T06:47:53&bounding_box=-24.697,63.281,-13.646,66.717&agent=NO&email=yes&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6

*For requests for data without subsetting, reformatting, or reprojection services (only spatial and/or temporal filtering), you must use the agent=NO parameter.


The programmatic access structure using curl is as follows:

curl <options> <endpoint>/request?<kvp operands separated with &>

Request example 1 using curl commands. In this example, the --dump-header option is used and the response-header.txt returns the HTTP response status code along with file name and/or debugging info. Data are downloaded in a single .zip file to the current working directory.

curl -O -J --dump-header response-header.txt "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=SPL3SMP&version=005&format=GeoTIFF&time=2018-06-06,2018-06-07&bounding_box=-109,37,-102,41&bbox=-109,37,-102,41&Coverage=/Soil_Moisture_Retrieval_Data_AM/
soil_moisture&projection=Geographic&email=yes&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6"

The following curl options may also be useful for programmatic access.

  -v       verbose, prints internal details of performing the curl request, used for debugging
  -s       silent, suppresses all unnecessary information output, used when not debugging
  -L       follow the referral to a new location, automatically handles the Code 303 redirect 
  -O       save the result in a local file
  -J       name the saved file according to the returned header, used with -O
  -F       used in POST command to upload file to endpoint
  -w       write specified info to stdout, used to capture return codes, file names and urls
  -i       show returned headers, useful for debugging
  --dump-header       include the HTTP header in the output, used to get the returned file name

Please see the Table of Key-Value-Pairs (KVPs) for information on available KVP operands for subsetting, reformatting, and reprojection services.

Guide to large file requests:

Request example: MOD10CM version 006, reformatting to GeoTIFF, subset /MOD_CMG_Snow_5km/Snow_Cover_Monthly_CMG parameter, from 2015-01-01 to 2017-01-01 (24 monthly files requested), with email requested:

curl -O -J --dump-header response-header.txt "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=MOD10CM&version=006&format=GeoTIFF&time=2015-01-01,2017-01-01&Coverage=/MOD_CMG_Snow_5km/Snow_Cover_Monthly_CMG&email=yes&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=true"

In this example, only 10 of the 24 files requested will be returned. This is due to the default page size of the CMR search results, which is set to 10 results. In order to receive your entire order in the case of 11 or more requested files, the page_num KVP is required to request successive CMR page results. The following set of bash commands loops through the page numbers until all files are returned from the example above:

i=1
while [ $? -eq 0 ]
do
    curl -O -J -L --dump-header response-header.txt "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=MOD10CM&version=006&time=2015-01-01,2017-01-01&Coverage=/MOD_CMG_Snow_5km/Snow_Cover_Monthly_CMG&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=true&page_num="$i
    i=$[$i+1]
    grep -E 'Content-Disposition: attachment; filename=".*zip"' response-header.txt
done

This loop will return multi-file zips for each page of CMR results (in this case, 3 zipped files are returned totaling 24 file outputs). 

Last updated: May 2019

86,5 Bot