In this section
  • Rendering Links ...
  • Rendering Links ...

How to access data using an FTP client, the command line, wget or Python

This tutorial describes how to access data via FTP using FTP client, command line, wget, and Python.

Note: Start by determining the FTP address associated with your desired dataset. This address is needed to specify the location of the data set or files of interest on the server. Below, you'll find the process for finding the FTP address, followed by instructions on how to access the data using the different methods: FTP client, command line, wget, and Python.

How to determine the FTP address for a data set

The FTP address varies depending on the data set and the FTP server where it is stored. Here are four ways to determine the FTP address:

  1. If you’ve subscribed to receive notifications about the data set, you can find this address in your data set registration confirmation email.
     
  2. If you are accessing Extreme Ice Survey (EIS) Glacier Image Archive, 2007-2022 data, the FTP address is ftp://ftp.nsidc.org/pub/DATASETS/nsidc0783_extreme_ice_survey_v1.
     
  3. If you are accessing Snow Today data, the FTP address is ftp://dtn23.rc.colorado.edu/shares/snow-today.
    The data are organized in directories by water year (WY####), then by data type. 
     
  4. If you are accessing GoLIVE data, the FTP link for the monthly processed archive data follows this pattern:
    ftp://dtn.rc.colorado.edu/work/nsidc0710/nsidc0710_landsat8_golive_ice_velocity_v1.1/pXXX_rYYY.
    For the near-real-time data, the FTP link pattern is: 
    ftp://dtn.rc.colorado.edu/work/nsidc0710/nsidc0710_landsat8_golive_ice_velocity_v1.1_nrt/pXXX_rYYY.
    In both cases 'pXXX_rYYY' refers to the path and row folder, for example, p001_r004. To determine these numbers, we recommend using the GoLIVE map application. Select the scenes in your region of interest,  and the path and row numbers will be displayed on the left-hand side bar. 
     
  5. If the dataset is available via FTP, you can find the FTP address from the dataset landing pages. In the ‘Data Access & Tools’ section, click ‘FTP File System’. This will open a new browser tab. Although most browsers have disabled FTP support and may display an error message, the FTP link in the search bar remains accurate. Copy the full FTP address. For instance, the full FTP address for the Circum-Arctic Map of Permafrost and Ground-Ice Conditions data set is ftp://sidads.colorado.edu/pub/DATASETS/fgdc/ggd318_map_circumarctic/.
     

How to access data using an FTP Client

To access data using an FTP client, such as FileZilla, Cyberduck, or FireFTP, please refer the following information:

Extreme Ice Survey (EIS) Glacier Image Archive, 2007-2022

Host Name: ftp.nsidc.org
User Name: anonymous
Password: <your e-mail address>
Directory: /pub/DATASETS/nsidc0783_extreme_ice_survey_v1

Snow Today

Host Name: dtn23.rc.colorado.edu
User Name: anonymous
Password: pwd
Directory: /shares/snow-today

GoLIVE

Host Name: dtn.rc.colorado.edu
User Name: anonymous
Password: <your e-mail address>
Directory: /work/nsidc0710

Data Sets on FTP

Host Name: sidads.colorado.edu
User Name: anonymous
Password: <your e-mail address>
Directory: /pub/DATASETS/XXXXXX
‘XXXXXX’ represents the subdirectory of the dataset, identifiable from the full FTP address you copied from the fourth option above. For instance, in the case of the Circum-Arctic Map of Permafrost and Ground-Ice Conditions dataset, XXXXXX is fgdc/ggd318_map_circumarctic/.

How to access data using the command line

Extreme Ice Survey (EIS) Glacier Image Archive, 2007-2022

  1. Connect to our node by typing the following:
    ftp ftp.nsidc.org
     
  2. On connecting, it will ask you to log in with a Name. Type:
    anonymous
     
  3. You should now be logged in. To navigate to the directory containing the data you need, type:
    cd /pub/DATASETS/nsidc0783_extreme_ice_survey_v1
     
  4. To see all the files in the folder, type:
    ls
     
  5. Set the transfer mode to binary by typing:
    binary
     
  6. To download a single file, type:
    get <filename>
     
  7. To log out, type:
    quit

Snow Today

  1. Connect to our node by typing the following:
    ftp dtn23.rc.colorado.edu
     
  2. On connecting, it will ask you to log in with a Name. Type:
    anonymous
     
  3. You should now be logged in, to navigate to the directory that contains the data you require, type:
    cd /shares/snow-today
     
  4. To see all the available data directories, type:
    ls
    You will need to continue changing directories (cd) until you get to the data files you want. The data are organized in directories by water year (WY####), then by data type.
     
  5. Set the transfer mode to binary by typing:
    binary
     
  6. To download a single file, type:
    get <filename>
     
  7. To log out, type:
    quit

GoLIVE

  1. Connect to our node by typing the following:
    ftp dtn.rc.colorado.edu
     
  2. On connecting, it will ask you to log in with a Name. Type:
    anonymous
     
  3. You should now be logged in, to navigate to a directory for a specific path and row for the archive data , type:
    cd work/nsidc0710/nsidc0710_landsat8_golive_ice_velocity_v1.1/pXXX_rYYY
    where 'pXXX_rYYY' refers to the path and row folder e.g., p001_r004. 
     
  4. To see all the files in the folder, type:
    ls
     
  5. Set the transfer mode to binary by typing:
    binary
     
  6. To download a single file, type:
    get <filename>
     
  7. To log out, type:
    quit

Data Sets on FTP

  1. Connect to our node by typing the following:
    ftp sidads.colorado.edu
     
  2. On connecting, it will ask you to log in with a Name. Type:
    anonymous
     
  3. You should now be logged in. To navigate to the directory containing the data you need, type:
    cd /DATASETS/XXXX
    Here 'XXXX' is the specific data set directory and any subfolders within that. 
     
  4. To see all the files in the folder, type:
    ls
     
  5. Set the transfer mode to binary by typing:
    binary
     
  6. To download a single file, type:
    get <filename>
     
  7. To log out, type:
    quit

How to access data using wget

Extreme Ice Survey (EIS) Glacier Image Archive, 2007-2022

To download a single file from FTP using wget you can use the following command:

wget --ftp-user=anonymous ftp://ftp.nsidc.org/pub/DATASETS/nsidc0783_extreme_ice_survey_v1/<path to folder and file you wish to download>

Snow Today

To download a single file from FTP using wget you can use the following command:

wget --ftp-user=anonymous ftp://dtn23.rc.colorado.edu/shares/snow-today/<path to folder and file you wish to download>

GoLIVE

To download a single file from FTP using wget you can use the following command:

wget --ftp-user=anonymous ftp://dtn.rc.colorado.edu/work/nsidc0710/<path to folder and file you wish to download>

Data Sets on FTP

To download a single file from FTP using wget, use the following command:

wget --ftp-user=anonymous <ftp://sidads.colorado.edu/DATASETS/><path to file>


To download multiple files, use the following command:

wget --ftp-user=anonymous -i <text file with FTP links>

Here,  <text file with FTP links> is a .txt file containing a list of FTP links to the files you want to download.


If you want to download all the files in one directory, use this command:

wget --ftp-user=anonymous -r -nd ftp://sidads.colorado.edu/DATASETS/<path to data set directory>

For example, to download all files from the Circum-Arctic Map of Permafrost and Ground-Ice Conditions data set, use:

wget --ftp-user=anonymous -r -nd ftp://sidads.colorado.edu/DATASETS/fgdc/ggd318_map_circumarctic

How to access data using Python 

The sample script provided below can be used to download all files from a directory on the FTP server. To use it, copy the script into your own file (for instance,  FTP_download.py), then modify the first three variables as needed before running the script. This script is written for accessing data sets available on FTP. However, users of Snow Today or GoLIVE data may be able to customize the script for their downloads. 

#!/usr/bin/env python
# NSIDC
# Sample script to download all the files within one directory on the FTP server
#
# Requires Python3 and the ftplib and os libraries

from ftplib import FTP
import os

### The following 3 variables can be changed ###
# 1. Set the directory you would like to download the files to
destdir='path to directory'

# 2. Set the path to the FTP directory that contains the data you wish to download.
# This example is for the Circum-Arctic Map of Permafrost and Ground-Ice Conditions data set 
# https://nsidc.org/data/ggd318
directory = '/DATASETS/fgdc/ggd318_map_circumarctic'

# 3. Set the password which will be your email address
password = 'email_address'

############################################
### Don't need to change this code below ###
############################################
# FTP server
ftpdir = 'sidads.colorado.edu'

#Connect and log in to the FTP
print('Logging in')
ftp = FTP(ftpdir)
ftp.login('anonymous',password)

# Change to the directory where the files are on the FTP
print('Changing to '+ directory)
ftp.cwd(directory)

# Get a list of the files in the FTP directory
files = ftp.nlst()
files = files[2:]
print(files)

#Change to the destination directory on own computer where you want to save the files
os.chdir(destdir)

#Download all the files within the FTP directory
for file in files:
    print('Downloading...' + file)
    ftp.retrbinary('RETR ' + file, open(file, 'wb').write)

#Close the FTP connection
ftp.quit()

 

Last Updated March 2024