How do I programmatically request data services (e.g. subsetting, reformatting, reprojection) using an API?

Last updated: 23 March 2017

The subsetting, reformatting, and reprojection services provided by NSIDC through NASA Earthdata Search can also be accessed programmatically as a synchronous REST interface. This programmatic access is provided via an HTTPS URL containing a series of key-value-pairs (KVPs), allowing you to request data constrained by spatial and temporal filters. The resulting output data are returned as either a single file or a multi-file zip. This functionality is currently being enhanced to support Web Coverage Service (WCS) compatibility. 

This article provides a step-by-step getting started guide to utilizing this API. In addition to this guide, comprehensive documentation provided by the API developers, as well as an FAQ document, are attached. For questions about accessing NSIDC data using this API, please contact NSIDC User Services: nsidc@nsidc.org 

Step 1: Determine the services enabled for your data set(s) of interest

The data services provided by NSIDC apply to MODIS, SMAP, AMSR-E, and ICESat/GLAS data, and are unique to each of these data sets. Three resources are provided below to help you gain information on dataset-specific subsetting, reformatting, and reprojection services.

A. The following tables list the services provided for each collection:

  1. SMAP
  2. MODIS
  3. ICESat/GLAS
  4. AMSR-E

B. Use Earthdata Search to navigate to the data set service form of interest:

  1. Navigate to NASA Earthdata Search. Enter the collection (data set) short name of interest in the search bar and click on a collection (multiple versions may be present).                                                                                                              
  2. On the next page, click on a file download button.                                                                                                   

  3. You will then be prompted to enter your Earthdata Login credentials. On the Data Access page, click on the short_name.version ESI Service radio button. This option will present all services available for that data set. Under "Band Subsetting (Optional)", all parameters contained within the file are displayed upon clicking the expandable tree.                                                                               

C. Query the capabilities service for your data set of interest:

Usage:

https://n5eil01u.ecs.nsidc.org/egi/capabilities/(SHORT_NAME).(VERSION).xml

This will return an xml file detailing the format, projection, and subsetting options, as well as the variable names contained within the data files. This can be a cumbersome file however, so the following command line example uses grep to return only the services of interest:

Request example: I want to know the reformatting options available for SPL3SMP version 004. 

curl "https://n5eil01u.ecs.nsidc.org/egi/capabilities/SPL3SMP.004.xml" | grep "Format value"

This will return the available formats to be used in Step 3 (use the format value as opposed to the label, if different):

<Format value="" label="No Reformatting" />
<Format value="NetCDF4-CF" label="NetCDF4-CF" />
<Format value="ASCII" label="ASCII" />
<Format value="KML" label="KML" />
<Format value="GeoTIFF" label="GeoTIFF" />
<Format value="NetCDF-3" label="NetCDF-3" />

The following terms can be grepped to determine additional services:

"Format value"

Reformatting options available for the data set. No results indicate that reformatting is unavailable.

"SubsetVariable value"

Parameter names contained within the data file to be used for parameter subsetting. No results indicate that parameter subsetting is unavailable.

"spatialSubsetting"

A line including either spatialSubsetting="true" or spatialSubsetting="false" is returned, indicating whether or not spatial subsetting is enabled. 

"Projection value"

Returns the available reprojection options. No results indicate that reprojection is unavailable.

"Projectionparameter"

Returns required and/or optional parameters associated with reprojection options.

"ResampleDimension value"

Returns the available resampling options. No results indicate that resampling is unavailable.

"InterpolationMethod"

Returns the available interpolation options. No results indicate that interpolation is unavailable.


Step 2: Create Token

A token is needed in order to access data using your Earthdata Login credentials. The token is valid for a limited period of time determined by Earthdata Login. See the following Earthdata Wiki article for more information on this step, including troubleshooting information and an alternative file input option: Creating a Token Common.  

curl example including NSIDC Provider ID (replace all items in bold with your personal credentials):

curl -X POST --header "Content-Type: application/xml" -d '<token><username>earthdata_login_user_name</username><password>earthdata_login_password</password><client_id>client_name_of_your_choosing</client_id><user_ip_address>your_origin_ip_address</user_ip_address> <provider>NSIDC_ECS</provider></token>' https://api.echo.nasa.gov/echo-rest/tokens

You should receive the token ID in the response. The value in the ID tag is the token you will use in Step 3. For example:

<token>
<id>75E5CEBE-6BBB-2FB5-A613-0368A361D0B6
</id>
<username>earthdata_login_user_name</username>
<client_id>client_name_of_your_choosing</client_id>
<user_ip_address>your_origin_ip_address</user_ip_address>
<provider>NSIDC_ECS</provider>
</token>

Step 3: Create the API endpoint. This includes the data host, KVPs used to search the Common Metadata Repository (CMR) for requested files, KVPs used to request data services, and KVPs to access Earthdata login information via your token from Step 2. 

The programmatic access structure is as follows:

https://n5eil01u.ecs.nsidc.org/egi/request?search_kvp=search_kvp&service_kvp=service_kvp&token=token&email=email

Request example: SMAP L3 Radiometer Global Daily 36 km EASE-Grid Soil Moisture, Version 4, GeoTIFF reformatting, parameter subsetting, and Geographic reprojection for all data collected 2016-12-11 to 2016-12-15:

https://n5eil01u.ecs.nsidc.org/egi/request?short_name=SPL3SMP&version=004&format=GeoTIFF&time=2016-12-11,2016-12-15&Coverage=/Soil_Moisture_Retrieval_Data_AM/soil_moisture&projection=Geographic&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=name@domain.com

The programmatic access structure using curl is as follows:

curl <options> <endpoint>/request?<parameters separated with &>

Request example using curl commands. In this example, the --dump-header option is used and the response-header.txt returns the HTTP response status code along with file name and/or debugging info. Data are downloaded in a single .zip file to the current working directory.

curl -O -J --dump-header response-header.txt "https://n5eil01u.ecs.nsidc.org/egi/request?short_name=SPL3SMP&version=004&format=GeoTIFF&time=2016-12-11,2016-12-15&Subset_Data_layers=/Soil_Moisture_Retrieval_Data_AM/soil_moisture&projection=Geographic&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=name@domain.com"    

The following curl options may also be useful for programmatic access. Please see the attached documentation for further guidance on curl.

  -v       verbose, prints internal details of performing the curl request, used for debugging
  -s       silent, suppresses all unnecessary information output, used when not debugging
  -L       follow the referral to a new location, automatically handles the Code 303 redirect 
  -O       save the result in a local file
  -J       name the saved file according to the returned header, used with -O
  -w       write specified info to stdout, used to capture return codes, file names and urls
  -i       show returned headers, useful for debugging
  --dump-header       include the HTTP header in the output, used to get the returned file name

The following table lists important KVPs associated with the Common Metadata Repository (CMR) file search, as well as all KVPs available for subsetting, reformatting, and reprojection services. Please see the CMR Search API Documentation page for additional information on CMR parameters. 

Key-value-pair operand Description Example

short_name

Required field. Specifies the short name ID of the data set requested.

Can be used multiple times to return files from multiple data sets.  

short_name=MOD10A1

email Required field. Specifies an email address to be used for contact information. 

email=name@domain.com

version

Specifies the data set version number. If not specified, all available versions will be returned. Please note that this is usually a 3-digit number but may also be 1 digit. To determine the correct verion number, query CMR collection metadata.
For example:

curl "https://cmr.earthdata.nasa.gov/search/collections.json?short_name=SPL3SMP&pretty=true" | grep version_id

version=004

time

Specifies a temporal range filter in start_time,end_time format.
Valid options include:

time=yyyy-MM-dd,yyyy-MM-dd
time=yyyy-MM-ddTHH:mm:ssZ,yyyy-MM-ddTHH:mm:ssZ
time=yyyy-MM-ddTHH:mm:ss,yyyy-MM-ddTHH:mm:ss

If the first option is used (no time specified), the day defaults to 
T00:00:00. This parameter is the WCS compatible equivalent of
the CMR 'temporal' parameter.

time=2016-12-11,2016-12-15

bounding_box

Specifies a search filter to find files with a spatial extent that
overlaps this bounding box, specified in decimal degrees of
latitude and longitude in WSEN format:

bounding_box=lower_left_long,lower_left_lat,
upper_right_long,upper_right_lat

bounding_box=-109,35,-105,38

sort_key

Specifies the sort order of returned files from the CMR search. If a temporal range search is performed, the search results will be sorted by temporal overlap percentage over all ranges provided by default. Otherwise, results are sorted by ascending entry title by default. See Sorting Collection Results in the CMR documentation for more info. 

Sort files by descending start date (most recent data first):

sort_key=-start_date

page_num

This is a CMR parameter to select the page of results to be processed. See Guide to large file requests below for more guidance on this parameter.

page_num=2

format

Specifies the output reformatting option. Supported options are data set-specific
(see Step 1 to determine valid format options). Options include:

[GeoTIFF, HDF-EOS5, NetCDF4-CF, NetCDF-3, ASCII, HDF-EOS, KML]

This parameter is optional; native file format will be returned if format
is not specified. 

format=GeoTIFF

coverage

Specifies the data set parameter (i.e. variable, layer) or group of parameters to be subsetted.
Multiple datasets can be specified separated by commas. Also signifies WCS coverage to be
processed. If only a group or subgroup is specified, all lower level datasets are included in the
processing. See Step 1 to help determine parameters contained within a data file.

coverage=/group/sub-group/sub-sub-group/parameter

[MOD10A1 example]
coverage=/MOD_Grid_Snow_500m/NDSI

bbox

Unlike bounding_box above, bbox specifies a bounding box to be used for spatial subsetting.
Output files will be cropped to the specified bounding box extent.

bbox=lower_left_long,lower_left_lat,upper_right_long,upper_right_lat

bbox=-109,35,-105,38

projection

Specifies the output reprojection option. Supported projections are dataset-specific 
(see Step 1 to determine valid projection options). Options include:

[geographic, universal+transverse+mercator, utm+northern+hemisphere,
utm+southern+hemisphere, north+polar+stereographic,
south+polar+stereographic, polar+stereographic, state+plane,
lambert+conformal+conic, transverse+mercator, lambert+azimuthal, sinusoidal]

Note that some projection options include additional required and optional parameters.
KVP Operand: projection_parameters

Projection_
Parameters

Applicable projection Description Optional or required

NZone:NN 

universal+transverse+
mercator

UTM zone number. Valid range=-60 to 60; <0 signifies southern hemisphere. Example:

NZone:-5

Not required; unavailable if bbox is specified (UTM zone assigned using center of bounding box)

LonZ:lon,
LatZ:lat

universal+transverse+
mercator

Latitude and longitude of any point in UTM Zone in decimal degrees. Example:

LonZ:-110,LatZ:45

Not required; unavailable if bbox is specified  (UTM zone assigned using center of bounding box)

NZone:NN

utm+northern+hemisphere
utm+southern+hemisphere

UTM zone number. Valid range=1 to 60. Example:

NZone:30

Required

NZone:NNNN

state+plane

State Plane zone code. Example:

NZone:0502

Not required; unavailable if bbox is specified  (State Plane zone assigned using center of bounding box)

OriginLat:NN

lambert+conformal+conic
transverse+mercator

Latitude of false/natural origin; Angular values.

Not required; unavailable if bbox is specified

CentMer:NN

lambert+conformal+conic
transverse+mercator

Longitude of false origin / central meridian

Longitude of natural/false origin; Angular values.

Not required; unavailable if bbox is specified

STDPR1:NN

lambert+conformal+conic

Latitude of first standard parallel; Angular values.

Not required; unavailable if bbox is specified

STDPR2:NN

lambert+conformal+conic

Latitude of second standard parallel; Angular values.

Not required; unavailable if bbox is specified

FE:NN

lambert+conformal+conic
polar+stereographic
transverse+mercator
lambert+azimuthal
sinusoidal

False Easting; Linear values.

Not required; unavailable if bbox is specified

FN:NN

lambert+conformal+conic
polar+stereographic
transverse+mercator
lambert+azimuthal
sinusoidal

False Northing; Linear values.

Not required; unavailable if bbox is specified

SMajor:NN

lambert+conformal+conic
polar+stereographic
transverse+mercator

Semimajor radius of the ellipsoid axis; Linear values.

Not required; unavailable if bbox is specified

SMinor:NN

lambert+conformal+conic
polar+stereographic
transverse+mercator

Semiminor radius of the ellipsoid axis; Linear values.

Not required; unavailable if bbox is specified

LongPol:NN

polar+stereographic

Longitude of natural origin; Angular values.

Not required; unavailable if bbox is specified

TrueScale:NN

polar+stereographic

Latitude of natural origin; Angular values.

Not required; unavailable if bbox is specified

Factor:N

transverse+mercator

Scale factor at natural origin; Unitless.

Not required; unavailable if bbox is specified

Sphere:NN

lambert+azimuthal

Radius of reference Sphere; Linear values. 

Not required; unavailable if bbox is specified

CenterLat:NN

lambert+azimuthal

Latitude of projection center / origin; Angular values.

Not required; unavailable if bbox is specified

CentLon:NN

lambert+azimuthal
sinusoidal

Longitude of projection center; Angular values. Not required; unavailable if bbox is specified

resample

Resample dimension, in percent. Valid range=1 to 200

resample=percent:NNN

resample=percent:50

interpolation

Interpolation option used in reprojection. Options include:

Interpolation Description

NN

Nearest Neighbor

BI

Bilinear

CC

Cubic Convolution

interpolation=NN

Guide to large file requests:

Request example: MOD10CM version 006, reformatting to GeoTIFF, subset /MOD_CMG_Snow_5km/Snow_Cover_Monthly_CMG parameter, from 2015-01-01 to 2017-01-01 (23 monthly files requested):

curl -O -J --dump-header response-header.txt "https://n5eil01u.ecs.nsidc.org/egi/request?short_name=MOD10CM&version=006&format=GeoTIFF&time=2015-01-01,2017-01-01&Coverage=/MOD_CMG_Snow_5km/Snow_Cover_Monthly_CMG&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=name@domain.com"

In this example, only 10 of the 23 files requested will be returned. This is due to:
1. The default page size of the CMR search results, and 
2. The NSIDC limit on number of files returned in one request, which can be determined by querying the capabilities service described in Step 1C:

curl "https://n5eil01u.ecs.nsidc.org/egi/capabilities/MOD10A1.006.xml" | grep maxGransSyncRequest

Both the file limit and default page size are set to 10 files. In order to receive your entire order in the case of 11 or more requested files, the page_num KVP is required to request successive CMR page results. The following set of bash commands loops through the page numbers until all files are returned from the example above:

i=1
while [ $? -eq 0 ]
do
    curl -O -J -L --dump-header response-header.txt 'https://n5eil01u.ecs.nsidc.org/egi/request?short_name=MOD10CM&version=006&time=2015-01-01,2017-01-01&Coverage=/MOD_CMG_Snow_5km/Snow_Cover_Monthly_CMG&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=name@domain.com&page_num='$i
    i=$[$i+1]
    grep -E 'Content-Disposition: attachment; filename=".*zip"' response-header.txt
done

 

This loop will return multi-file zips for each page of CMR results (in this case, 3 zipped files are returned totaling 23 file outputs).