Knowledge Base

How do I programmatically request data services such as subsetting, reformatting, and reprojection using an API?

The subsetting, reformatting, and reprojection services provided by NSIDC through NASA Earthdata Search can also be accessed programmatically as a synchronous REST interface. This programmatic access is provided via an HTTPS URL containing a series of key-value-pairs (KVPs), allowing you to request data constrained by spatial and temporal filters. The resulting output data are returned as either a single file or a multi-file zip. This functionality is currently being enhanced to support Web Coverage Service (WCS) compatibility. 

This article provides a step-by-step getting started guide to utilizing this API. In addition to this guide, comprehensive documentation provided by the API developers, as well as Frequently Asked Questions, can be found on the Earthdata Developer Portal.
For questions about accessing NSIDC data using this API, please contact NSIDC User Services: nsidc@nsidc.org

Step 1: Determine the services enabled for your data set(s) of interest

The data services provided by NSIDC apply to MODIS, SMAP, AMSR-E, and ICESat/GLAS data, and are unique to each of these data sets. 

A. The following tables list an overview of services provided for each collection:

  1. SMAP
  2. MODIS
  3. ICESat/GLAS
  4. AMSR-E

B. Detailed service capabilities can be queried for your data set of interest:

Usage:

https://n5eil01u.ecs.nsidc.org/egi/capabilities/(SHORT_NAME).(VERSION).xml

This will return an xml file detailing the format, projection, and subsetting options, as well as the variable names contained within the data files. This can be a cumbersome file, however, so the following command line example uses grep to return only the services of interest:

Request example: I want to know the reformatting options available for SPL3SMP version 004.

curl "https://n5eil01u.ecs.nsidc.org/egi/capabilities/SPL3SMP.004.xml" | grep "Format value"

This will return the available formats to be used in Step 3 (use the format value as opposed to the label, if different):

<Format value="" label="No Reformatting" />
<Format value="NetCDF4-CF" label="NetCDF4-CF" />
<Format value="ASCII" label="ASCII" />
<Format value="KML" label="KML" />
<Format value="GeoTIFF" label="GeoTIFF" />
<Format value="NetCDF-3" label="NetCDF-3" />

The following terms can be grepped to determine additional services:

"Format value"

Reformatting options available for the data set. No results indicate that reformatting is unavailable.

"SubsetVariable value"

Parameter names contained within the data file to be used for parameter subsetting. No results indicate that parameter subsetting is unavailable.

"spatialSubsetting"

A line including either spatialSubsetting="true" or spatialSubsetting="false" is returned, indicating whether or not spatial subsetting is enabled.

"Projection value"

Returns the available reprojection options. No results indicate that reprojection is unavailable.

"Projectionparameter"

Returns required and/or optional parameters associated with reprojection options.

"ResampleDimension value"

Returns the available resampling options. No results indicate that resampling is unavailable.

"InterpolationMethod"

Returns the available interpolation options. No results indicate that interpolation is unavailable.

Step 2: Create Token

A token is needed in order to access data using your Earthdata Login credentials. The token is valid for 30 days. See the following Earthdata Wiki article for more information on this step, including troubleshooting information and an alternative file input option: Creating a Token Common.

curl example including NSIDC Provider ID (replace all items in bold with your personal credentials):

curl -X POST --header "Content-Type: application/xml" -d '<token><username>earthdata_login_user_name</username><password>earthdata_login_password</password><client_id>client_name_of_your_choosing</client_id><user_ip_address>your_origin_ip_address</user_ip_address> <provider>NSIDC_ECS</provider></token>' https://api.echo.nasa.gov/echo-rest/tokens

You should receive the token ID in the response. The value in the ID tag is the token you will use in Step 3. For example:

<token>
<id>75E5CEBE-6BBB-2FB5-A613-0368A361D0B6</id>
<username>earthdata_login_user_name</username>
<client_id>client_name_of_your_choosing</client_id>
<user_ip_address>your_origin_ip_address</user_ip_address>
<provider>NSIDC_ECS</provider>
</token>

Step 3: Create the API endpoint. This includes the data host, KVPs used to search the Common Metadata Repository (CMR) for requested files, KVPs used to request data services, and KVPs to access Earthdata login information via your token from Step 2.

The programmatic access structure is as follows:

https://n5eil01u.ecs.nsidc.org/egi/request?search_kvp=search_kvp&service_kvp=service_kvp&token=token&email=email

Request example: SMAP L3 Radiometer Global Daily 36 km EASE-Grid Soil Moisture, Version 4, GeoTIFF reformatting, parameter subsetting, and Geographic reprojection for all data collected 2016-12-11 to 2016-12-15:

https://n5eil01u.ecs.nsidc.org/egi/request?short_name=SPL3SMP&version=004&format=GeoTIFF&time=2016-12-11,2016-12-15&Coverage=/Soil_Moisture_Retrieval_Data_AM/soil_moisture&projection=Geographic&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=name@domain.com

The programmatic access structure using curl is as follows:

curl <options> <endpoint>/request?<parameters separated with &>

Request example using curl commands. In this example, the --dump-header option is used and the response-header.txt returns the HTTP response status code along with file name and/or debugging info. Data are downloaded in a single .zip file to the current working directory.

curl -O -J --dump-header response-header.txt "https://n5eil01u.ecs.nsidc.org/egi/request?short_name=SPL3SMP&version=004&format=GeoTIFF&time=2016-12-11,2016-12-15&Subset_Data_layers=/Soil_Moisture_Retrieval_Data_AM/soil_moisture&projection=Geographic&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=name@domain.com"

The following curl options may also be useful for programmatic access. Please see the attached documentation for further guidance on curl.

  -v       verbose, prints internal details of performing the curl request, used for debugging
  -s       silent, suppresses all unnecessary information output, used when not debugging
  -L       follow the referral to a new location, automatically handles the Code 303 redirect 
  -O       save the result in a local file
  -J       name the saved file according to the returned header, used with -O
  -w       write specified info to stdout, used to capture return codes, file names and urls
  -i       show returned headers, useful for debugging
  --dump-header       include the HTTP header in the output, used to get the returned file name

The following table lists important KVPs associated with the Common Metadata Repository (CMR) file search, as well as all KVPs available for subsetting, reformatting, and reprojection services. Please see the CMR Search API Documentation page for additional information on CMR parameters.

Key-value-pair operand Description Example

short_name

Required field. Specifies the short name ID of the
data set requested.

Can be used multiple times to return files from multiple
data sets.

short_name=MOD10A1

email Required field. Specifies an email address to be used
for contact information.

email=name@domain.com

version

Specifies the data set version number. If not specified,
all available versions will be returned. Please note that
this is usually a 3-digit number but may also be 1 digit.
To determine the correct verion number, query CMR
collection metadata.
For example:

curl "https://cmr.earthdata.nasa.gov/search/
collections.json?short_name=SPL3SMP&pretty=true"
| grep version_id

version=004

time

Specifies a temporal range filter in start_time,end_time
format. Valid options include:

time=yyyy-MM-dd,yyyy-MM-dd
time=yyyy-MM-ddTHH:mm:ssZ,yyyy-MM-ddTHH:mm:ssZ
time=yyyy-MM-ddTHH:mm:ss,yyyy-MM-ddTHH:mm:ss

If the first option is used (no time specified), the day
defaults to T00:00:00. This parameter is the WCS
compatible equivalent of the CMR 'temporal'
parameter.

time=
2016-12-11,2016-12-15

bounding_box

Specifies a search filter to find files with a spatial extent
that overlaps this bounding box, specified in decimal
degrees of latitude and longitude in WSEN format:

bounding_box=lower_left_long,lower_left_lat,
upper_right_long,upper_right_lat

bounding_box=
-109,35,-105,38

sort_key

Specifies the sort order of returned files from the CMR
search. If a temporal range search is performed, the
search results will be sorted by temporal overlap
percentage over all ranges provided by default.
Otherwise, results are sorted by ascending entry title by
default. See Sorting Collection Results in the CMR
documentation for more info.

Sort files by descending start date (most recent data first):

sort_key=-start_date

page_num

This is a CMR parameter to select the page of results to
be processed. See Guide to large file requests below
for more guidance on this parameter.

page_num=2

format

Specifies the output reformatting option. Supported options
are data set-specific (see Step 1 to determine valid format
options). Options include:

[GeoTIFF, HDF-EOS5, NetCDF4-CF, NetCDF-3, ASCII, HDF-EOS,
KML]


This parameter is optional; native file format will be returned
if format is not specified.

format=GeoTIFF

coverage

Specifies the data set parameter (i.e. variable, layer) or group
of parameters to be subsetted. Multiple datasets can be
specified separated by commas. Also signifies WCS coverage
to be processed. If only a group or subgroup is specified, all
lower level datasets are included in the processing. See Step 1
to help determine parameters contained within a data file.

coverage=/group/sub-group/sub-sub-group/parameter

[MOD10A1 example]
coverage=
/MOD_Grid_Snow_500m/NDSI

bbox

Unlike bounding_box above, bbox specifies a bounding box to
be used for spatial subsetting. Output files will be cropped to
the specified bounding box extent.

bbox=lower_left_long,lower_left_lat,
upper_right_long,upper_right_lat

bbox=-109,35,-105,38

projection

Specifies the output reprojection option. Supported projections
are dataset-specific (see Step 1 to determine valid projection
options). Options include:

[geographic, universal+transverse+mercator,
utm+northern+hemisphere,utm+southern+hemisphere,
north+polar+stereographic, south+polar+stereographic,
polar+stereographic, state+plane, lambert+conformal+conic,
transverse+mercator, lambert+azimuthal, sinusoidal]

Note that some projection options include additional required and
optional parameters.
KVP Operand: projection_parameters

Projection_
Parameters

Applicable
projection

Description Optional or
required

NZone:NN 

universal+
transverse+
mercator

UTM zone number.
Valid range=
-60 to 60;
<0 signifies southern hemisphere. Example:

NZone:-5

Not required; unavailable if bbox is specified (UTM zone assigned using center of bounding box)

LonZ:lon,
LatZ:lat

universal+
transverse+
mercator

Latitude and longitude
of any point in UTM
Zone in decimal degrees. Example:

LonZ:-110,
LatZ:45

Not required; unavailable if bbox is specified (UTM zone assigned using center of bounding box)

NZone:NN

utm+northern+
hemisphere

utm+southern+
hemisphere

UTM zone number. Valid range=1 to 60. Example:

NZone:30

Required

NZone:NNNN

state+plane

State Plane zone code. Example:

NZone:0502

Not required; unavailable if bbox is specified (State Plane zone assigned using center of bounding box)

OriginLat:NN

lambert+
conformal+
conic

transverse+
mercator

Latitude of false/natural
origin; Angular values.

Not required; unavailable if bbox is specified

CentMer:NN

lambert+
conformal+
conic

transverse+
mercator

Longitude of false origin / central meridian

Longitude of natural/false origin; Angular values.

Not required; unavailable if bbox is specified

STDPR1:NN

lambert+
conformal+
conic

Latitude of first standard parallel; Angular values.

Not required; unavailable if bbox is specified

STDPR2:NN

lambert+
conformal+
conic

Latitude of second
standard parallel;
Angular values.

Not required; unavailable if bbox is specified

FE:NN

lambert+
conformal+
conic

polar+
stereographic

transverse+
mercator

lambert+
azimuthal

sinusoidal

False Easting;
Linear values.

Not required; unavailable if bbox is specified

FN:NN

lambert+
conformal+
conic

polar+
stereographic

transverse+
mercator

lambert+
azimuthal

sinusoidal

False Northing;
Linear values.

Not required; unavailable if bbox is specified

SMajor:NN

lambert+
conformal+
conic

polar+
stereographic

transverse+
mercator

Semimajor radius of
the ellipsoid axis; Linear values.

Not required; unavailable if bbox is specified

SMinor:NN

lambert+
conformal+
conic

polar+
stereographic

transverse+
mercator

Semiminor radius of
the ellipsoid axis; Linear values.

Not required; unavailable if bbox is specified

LongPol:NN

polar+
stereographic

Longitude of natural
origin; Angular values.

Not required; unavailable if bbox is specified

TrueScale:NN

polar+
stereographic

Latitude of natural
origin; Angular values.

Not required; unavailable if bbox is specified

Factor:N

transverse+
mercator

Scale factor at natural
origin; Unitless.

Not required; unavailable if bbox is specified

Sphere:NN

lambert+
azimuthal

Radius of reference
Sphere; Linear values.

Not required; unavailable if bbox is specified

CenterLat:NN

lambert+
azimuthal

Latitude of projection
center / origin;
Angular values.

Not required; unavailable if bbox is specified

CentLon:NN

lambert+
azimuthal

sinusoidal

Longitude of projection
center; Angular values.
Not required; unavailable if bbox is specified

resample

Resample dimension, in percent. Valid range=1 to 200

resample=percent:NNN

resample=percent:50

interpolation

Interpolation option used in reprojection. Options include:

Interpolation Description

NN

Nearest Neighbor

BI

Bilinear

CC

Cubic Convolution

interpolation=NN

Guide to large file requests:

Request example: MOD10CM version 006, reformatting to GeoTIFF, subset /MOD_CMG_Snow_5km/Snow_Cover_Monthly_CMG parameter, from 2015-01-01 to 2017-01-01 (23 monthly files requested):

curl -O -J --dump-header response-header.txt "https://n5eil01u.ecs.nsidc.org/egi/request?short_name=MOD10CM&version=006&format=GeoTIFF&time=2015-01-01,2017-01-01&Coverage=/MOD_CMG_Snow_5km/Snow_Cover_Monthly_CMG&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=name@domain.com"

In this example, only 10 of the 23 files requested will be returned. This is due to the default page size of the CMR search results, which is set to 10 results. In order to receive your entire order in the case of 11 or more requested files, the page_num KVP is required to request successive CMR page results. The following set of bash commands loops through the page numbers until all files are returned from the example above:

i=1
while [ $? -eq 0 ]
do
    curl -O -J -L --dump-header response-header.txt 'https://n5eil01u.ecs.nsidc.org/egi/request?short_name=MOD10CM&version=006&time=2015-01-01,2017-01-01&Coverage=/MOD_CMG_Snow_5km/Snow_Cover_Monthly_CMG&token=75E5CEBE-6BBB-2FB5-A613-0368A361D0B6&email=name@domain.com&page_num='$i
    i=$[$i+1]
    grep -E 'Content-Disposition: attachment; filename=".*zip"' response-header.txt
done

This loop will return multi-file zips for each page of CMR results (in this case, 3 zipped files are returned totaling 23 file outputs). 

Last updated: 20 November 2017

86,5 Bot