Hierarchical Data Format - Earth Observing System (HDF-EOS)



Converting from HDF to Binary Format


When converting from HDF to binary format, you can use either the hdp utility or the IDL program hdfeos2bin.pro. This Web page contains procedures for both methods.


Table of Contents


Use the following links to access the procedures on this Web page:

Converting from HDF to Binary Format Using the hdp Utility

UNIX/Linux Users

Windows NT/98/2000/XP Users

Converting from HDF to Binary Format Using IDL hdfeos2bin.pro


Converting from HDF to Binary Format Using the hdp Utility

The HDF Group provides various utilities for viewing the contents of Hierarchical Data Format (HDF) files and extracting the raster, binary, or ASCII objects. The hdp utility provides a simple way to dump HDF objects into a binary or ASCII format. There are several commands that can be used with hdp which can be obtained by typing:

hdp

The command used in these instructions is dumpsds, which displays data contained in a Scientific Data Set (SDS). The HDF Group Web site has documentation for using hdp, but it is essentially the same information one would obtain from typing:

hdp -h <command>

We provide more comprehensive yet simple instructions for dumping HDF objects to binary format.


UNIX/Linux Users


Download HDF Libraries from The HDF Group Web Site

HDF libraries are required to run hdp on UNIX/Linux platforms. The following instructions assume you are using either sh, csh, or tcsh on a UNIX/Linux system.

  1. Go to the szip compression external libraries pre-compiled binaries at:
    ftp://ftp.hdfgroup.org/lib-external/szip/2.0/bin/.

  2. Follow the link to the directory for your operating system and download the corresponding "noenc" file to your home directory.
    Example: szip2.0-linux-noenc.tar.gz

  3. Gunzip the file you downloaded using the following command:

    gunzip szip2.0-linux-noenc.tar.gz

  4. Untar the resulting tar file using the following command:

    tar xvf szip2.0-linux-noenc.tar

  5. Note: This creates the directory szip2.0-linux-noenc in your home directory.

  6. Check to see if you have the environment variable LD_LIBRARY_PATH defined by typing:

    echo $LD_LIBRARY_PATH

  7. Choose one of the following steps:
  1. If the environment variable LD_LIBRARY_PATH is defined, add the szip library directory to your existing LD_LIBRARY_PATH environment variable by typing:

    setenv LD_LIBRARY_PATH LD_LIBRARY_PATH\:$HOME/szip2.0-linux-noenc/lib

  2. If the environment variable LD_LIBRARY_PATH is not defined, the following message displays on your screen: LD_LIBRARY_PATH:Undefined variable, set the LD_LIBRARY_PATH environment variable to the szip library directory by typing:

    setenv LD_LIBRARY_PATH $HOME/szip2.0-linux-noenc/lib

Download hdp from The HDF Group Web Site

  1. Go to Pre-Compiled Binary Distributions.

  2. Select the appropriate directory for your platform.

  3. Follow the link to the utilities directory.

  4. Download hdp to your local drive using an ftp application.

  5. Type the following command to ensure the owner and the group of the file have read, write, and execute permission for running hdp:

  6. chmod 775 hdp

Identify HDF Variable Names

Users may chose to dump all HDF content to binary or just the contents of a single SDS. The names of the SDSs are listed as Variable Names in the HDF header.

  1. Dump the header information and print only the variable names to the screen by typing:

    hdp dumpsds -h <inputfilename.hdf> | grep <variable name>

    Example: Variable names are displayed in the following format:

    Variable Name = Latitude
    Variable Name = Longitude

  2. Take note of the variable name you wish to dump to binary format.

Dump HDF to Binary Format

  1. Type the following command:

    hdp dumpsds -n <variable name> -o <outputfilename> -b <inputfilename.hdf>

    Note 1: Use double quotes around the variable name if it consists of two or more words, for example "Sea Ice by Reflectance."

    Note 2: Byte order is big-endian for Sun/Solaris, SGI, HP, and IBM UNIX platforms.

Windows NT/98/2000/XP Users


Download hdp from The HDF Group Web Site

  1. Go to: ftp://ftp.hdfgroup.org/HDF/HDF_Current/bin/windows/.

  2. Download the hdp.exe file to your local drive.

  3. For Windows XP users, you must download three library files, zlib1.dll, szlibdll.dll, and szlibdll.lib, to the same directory where you downloaded the hdp.exe file.

  4. Download zlib123

    Download szlibdll.dll and szlibdll.lib

  5. Choose one of the following steps to open a Microsoft Windows DOS command prompt window:
  1. On your PC, select Start --> Run --> type the word Command in the dialog box.

  2. On your PC, select Start --> All Programs --> Accessories --> Command prompt.
  1. Change directories in the command prompt window until you are in the directory with the hdp executable. Be sure the HDF file is also in the same directory.

Identify HDF Variable Names

Users may choose to dump all HDF content to binary or just the contents of a single SDS. The names of the SDSs are listed as Variable Names in the HDF header.

  1. Dump the header information to an output file by typing the following command:

    hdp dumpsds -h -o <outputfilename> <inputfilename.hdf>

  2. Using a text editor of your choice, open the output file and scroll to the end of the file to view the variable names.

  3. Take note of the variable name you wish to dump to binary format.

Dump HDF to Binary Format

  1. At the MS-DOS command line, type the following command:

    hdp dumpsds -n <variable name> -o <output filename> -b <inputfilename.hdf>

    Note 1: Use double quotes around the field name if it consists of two or more words, for example "Day Tile Snow Cover."

    Note 2: Byte order is little-endian for Windows PC platforms.

Example

The following example uses a MODIS/Terra Snow Cover Daily L3 Global 0.05Deg CMG (MOD10C1) Version 5 (V005) file in HDF-EOS format to illustrate the use of hdp to extract a single data array from an HDF file, and save it to a flat binary file.

Identify HDF Variable Names

  1. Download the following MOD10C1 file to the directory containing the hdp executable:

    MOD10C1.A2000062.005.2006254085832.hdf

    Note: You may be able to right-click the mouse over the file name to download it. However, do not left-click the file name because this will display the HDF code in your browser.

  2. Choose one of the following steps to dump the header information to determine the variable names:
  1. At a UNIX command line, type the following command:

    hdp dumpsds -h MOD10C1.A2000062.005.2006254085832.hdf | grep <variable name>

  2. At a Windows MS-DOS command line, typing the following command:

    hdp dumpsds –h –o output.text MOD10C1.A2000062.005.2006254085832.hdf
  1. Using a text editor of your choice, open the output file and scroll to the end of the file to view the variable names.


  2. Example: Variable names are displayed in the following format:

    Variable Name = Day_CMG_Snow_Cover
    Variable Name = Day_CMG_Confidence_Index
    Variable Name = Day_CMG_Cloud_Obscured
    Variable Name = Snow_Spatial_QA

  3. Take note of the variable name you wish to dump to binary format.

Dump HDF to Binary Format

  1. Type the following command to dump the Day_CMG_Snow_Cover variable to the following binary file, MOD10C1005_snowcover.out:

    hdp dumpsds –n Day_CMG_Snow_Cover –o MOD10C1005_snowcover.out –b MOD10C1.A2000062.005.2006254085832.hdf

  2. Download one of the following links, which contain the binary output file, to your local drive for comparison with your own output file.

    Note: You may be able to right-click the mouse over the file name to download it. However, do not left-click the file name because this will display the HDF code in your browser.

    MOD10C1005snowcover.out (PC version, little-endian byte order)

    MOD10C1005snowcover.out (UNIX version, big-endian byte order)

Converting from HDF to Binary Format Using IDL hdfeos2bin.pro

The IDL program hdfeos2bin.pro, written by Terry Haran of the National Snow and Ice Data Center, extracts data arrays from an HDF-EOS file and writes the arrays to separate flat binary data files. This program has been tested and works on all Moderate-Resolution Imaging Spectroradiometer (MODIS) products, all Advanced Microwave Scanning Radiometer - Earth Observing System (AMSR-E) products, except Level-1A and Level-2 Land, and the Near-Real-Time SSM/I EASE-Grid Daily Global Ice Concentration and Snow Extent (NISE) data, but was designed to work with any HDF-EOS file.

The program hdfeos2bin.pro calls several subprograms that read swath or grid input data from HDF-EOS files and writes the output data to flat binary files. While this program has limited capabilities, it effectively demonstrates the use of HDF-EOS library calls without the need for installing and linking to HDF and HDF-EOS libraries because IDL effectively provides these libraries with built-in functions and procedures. The hdfeos2bin.pro program contains detailed documentation within the code.


UNIX/Linux and Windows NT/98/2000/XP Users

  1. Download the IDL program hdfeos2bin.pro and the following subprograms to the directory containing the HDF-EOS files.

    Note: You may be able to right-click the mouse over the file name to download it. However, do not left-click the file name because this will display the HDF code in your browser.

    process_grid.pro
    process_grid_field.pro
    process_swath.pro
    process_swath_field.pro
    write_field.pro

  2. Choose one of the following steps:
  1. In UNIX/Linux, change the current working directory to the directory containing the HDF files before invoking IDL.

    Note: The IDL command printd displays the current directory.

  2. In Windows, launch IDL and then change the current working directory to the location of the HDF files by typing the following command:

    cd, ‘\home\HDF_Directory’
  1. Choose one of the following steps to compile the hdfeos2bin.pro program:
  1. For the verbose option, type the following command:

    hdfeos2bin, ‘hdfeos_file’, /verbose

  2. For the non verbose option, type the following command:

    hdfeos2bin, ‘hdfeos_file’

    Note: Where ‘hdfeos_file’ is the name of the HDF-EOS file to be read. You may also specify filenames with wild cards, for example ‘*hdf’.
  1. After hdfeos2bin.pro and its subprograms run, list the files in your current directory.

    Note 1: You will see a new directory with the name of the HDF-EOS file and a .dir extension. Within the new directory, there will be binary files for each data variable in the HDF-EOS file.

    Note 2: The binary file names include the data type and the dimensions of the binary arrays.

    Example: Variable_TT_CCCCC_RRRRR.img.

    Where:

    Variable = Data variable name

    TT = Data type (u1 = unsigned 1-byte integers; u2 = unsigned 2-byte integers; s2 = signed 2-byte integers; f4 = 4-byte, single-precision floating-point numbers)

    CCCCC = Number of columns

    RRRRR = Number of rows

Example

The following example uses the AMSR-E/Aqua 5-Day L3 Global Snow Water Equivalent EASE-Grids (AE_DySno) data set in HDF-EOS format. We illustrate the use of hdfeos2bin.pro to extract data arrays from the HDF-EOS file and write the arrays to separate flat binary files.

Identify HDF Variable Names


  1. Download the following AE_DySno file to the directory containing hdfeos2bin.pro.

    AMSR_E_L3_5DaySnow_B07_20070126.hdf

    Note: You may be able to right-click the mouse over the file name to download it. Do not left-click the file name, as this will display the HDF code in your browser.

  2. Perform one of the following steps:
  1. In UNIX/Linux, change the current working directory to the directory containing the HDF file before invoking IDL.

  2. In Windows, launch IDL and then change the current working directory to the location of the HDF file by typing the following command:

    cd, ‘\AMSR_example’ 
  1. Type the following command:

    hdfeos2bin, 'AMSR_E_L3_5DaySnow_B07_20070126.hdf'

  2. After hdfeos2bin.pro and its subprograms run, list the files in your current directory.

    Note: You will see a new directory called AMSR_E_L3_5DaySnow_B07_20070126.hdf.dir. In that directory, you will see the following binary files:

    Flags_NorthernPentad_u1_00721_00721.img
    Flags_SouthernPentad_u1_00721_00721.img
    SWE_NorthernPentad_u1_00721_00721.img
    SWE_SouthernPentad_u1_00721_00721.img

    Based on the file names, the binary files are unsigned 1-byte integers and have 721 columns by 721 rows.