Knowledge Base

Options Available for Bulk Downloading Data from HTTPS with Earthdata Login

There are a few options depending on the tools that you have available.
•    Using your browser (Firefox only)
•    WGET
•    cURL

Using your browser (Firefox only)
You can use the Download Them All plugin for Firefox if you are using a Mac or Windows computer:
Download Them All is not supported in Firefox Quantum, which is the newest verison of the browser.

1. In your Firefox browser, navigate to the directory containing files you want to download.
2. Right click anywhere in the window and select "DownThemAll!"
3. A window will appear that allows you to select which files to download.
4. The process runs in the background so you can go to another directory.

WGET Instructions - for command line in Mac and Unix/Linux
1. Configure your username and password for authentication using a .netrc file

echo "machine login <uid> password <password>" >> ~/.netrc
chmod 0600 ~/.netrc

where <uid> is your Earthdata Login username and <password> is your URS password. Do not include the brackets <>.

2. Use a WGET command to download your data. Example WGET command:

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --no-check-certificate --auth-no-challenge=on -r --reject "index.html*" -np -e robots=off < insert complete data HTTPS URL >

WGET Instructions - for Windows
1. Create a text file to store the website cookies returned from the HTTPS server, called "mycookies.txt". Store this in the wget installation directory.

2. Use a WGET command to download your data. Example WGET command:

wget --http-user=[USERNAME] --http-password=[PASSWORD] --load-cookies mycookies.txt --save-cookies mycookies.txt --keep-session-cookies --no-check-certificate--auth-no-challenge -r --reject "index.html*" -np -e robots=off <insert complete HTTPS data URL>

WGET Tips 

Mac and Unix/Linux users can also skip the step of storing the username and password in a netrc file by adding --http-user=[USERNAME] --http-password=[PASSWORD], or  --http-user=[USERNAME] --ask-password.

The WGET examples provided in this article will download files from the specified directory to a directory on your machine. The directory on your machine will have the title of the HTTPS host. For NSIDC, this will either be "" or “”.

You can modify how the files are saved using the using the following WGET flags:
-nd (or --no-directories)
-nH (or --no-host-directories)
--cut-dirs=number (where the number is the number of directories to cut, but doesn't include the host directory name - you'd need to include -nH to remove that still)

The GNU WGET 1.18 Manual provides more details on WGET options.

CURL Instructions - for command line in Mac and Unix/Linux
Step 1 of the WGET instructions still applies, just use the cURL command instead of WGET.

curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n -O <insert complete HTTPS data URL>

The -O option on the cURL command downloads the file to the current working directory on your computer. 

Last updated: 21 December 2017

86,5 Bot