NPP Aggregation Tool for JPSS Product Files (nagg)
Version 1.4.0
August 2012
Copyright Notice and
License Terms for HDF NPP Aggregation Tool (nagg)
HDF NPP Aggregation Tool (nagg)
Copyright 2011-2012 by The HDF Group.
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted for any purpose (including commercial purposes) provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions, and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions, and the following disclaimer in the documentation and/or materials provided with the distribution.
3. In addition, redistributions of modified forms of the source or binary code must carry prominent notices stating that the original code was changed and the date of the change.
4. All publications or advertising materials mentioning features or use of this software are asked, but not required, to acknowledge that it was developed by The HDF Group and credit the contributors.
5. Neither the name of The HDF Group nor the name of any Contributor may be used to endorse or promote products derived from this software without specific prior written permission from The HDF Group or the Contributor, respectively.
DISCLAIMER:
THIS SOFTWARE IS PROVIDED BY THE HDF GROUP "AS IS" WITH NO WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED. In no event shall The HDF Group or the Contributors be liable for any damages suffered by the users arising out of the use of this software, even if advised of the possibility of such damage.
This document provides a brief introduction to the nagg tool, its uses, how to build and install it, the nagg command syntax, and a variety of examples.
nagg is a tool for aggregating JPSS data granules from existing files into new files with a different number of granules per file or different combinations of compatible products than in the original files. The tool was created to provide for individual users the ability to rearrange NPP product data granules from downloaded files into new files with aggregations or packaging that are better suited as input for a particular application. Being able to rearrange granules means users only have to request and download data once.
nagg
is needed to address the flexibility of NPP products, especially swath
products. A swath is a ribbon of data collected as the satellite orbit sweeps
across the Earth. As an orbit is continuous, the swath is continuous, wrapping
around the Earth over and over again like a yarn on a ball. In most NASA
heritage missions, each swath is snipped into equal size pieces called
granules, and one granule is placed in each file. The size of the snipped piece,
while usually set for each mission, varies from mission to mission based mostly
on the amount of data and therefore the size of the file that can be
conveniently manipulated.
For NPP, the
concept of a granule is independent of the packaging of the granule in a file,
so granules are made relatively small in order to conveniently manipulate them.
Suppose an analyst needs to compare MODIS and VIIRS data. With nagg, four VIIRS 86 second granules can be
packaged together to make the piece of swath similar in size to the MODIS 5
minute data granule. Some data analysis tools analyze by full orbit, so it is possible
to package OMPS data that way. nagg makes it possible for the analyst to package
NPP data into files in the way that best suits the tool being used.
The following is a list of naggÕs current and future capabilities:
á Copies data granules from input files to new files with the number of data granules per file specified by the –n command line option. The number data granules copied can range from 1 to 10,000.
á Copies data granules from input files to new files with the number of seconds of data provided by the granules in each file specified by the –A command line option. nagg will convert the number of seconds specified to the smallest integer number of data granules that will provide the seconds of data specified.
á Creates output files in the current directory or in a directory specified by the –d command line option.
á Copies NPP data product granules and corresponding geolocation product granules from separate input files to one output file.
á Copies NPP data product granules and corresponding geolocation product granules from one input file to separate output files.
á Copies data granules from separate input files to new output files containing more than one compatible NPP product. Compatible products share geolocation data, which is optional for inclusion, but if geolocation data is included it will be copied to the output file.
á Copies data granules from one input file containing compatible NPP products to separate output files for each product. Geolocation data may be excluded.
á Copies geolocation granules corresponding to sensor data products to new geolocation files or groups. If geolocation granules are not available or unwanted, the product granules may be aggregated or packaged without geolocation with the -g no command line option.
á Copies geolocation granules alone to new geolocation files without product granules using the -g <product> option.
á nagg always copies existing data granules from input data files to new data files that it creates: no changes are made to input files.
See the ÒCommand Line Options and ParametersÓ section on page 10 for more information.
The following are some notes regarding output file names, fill granules, and bucket boundaries.
á Output file names are determined by the products and timestamps of the granules in the file, the creation time of the file, and the –O and –D command line options. See section 3.4.1 ÒFile Naming Convention for NPP/NPOESS Data ProductsÓ in the JPSS Common Data Format Control Book – External Volume I, p. 22 for more information.
á Fill granules are added whenever there is a gap larger than the size of a granule between two input granules. Fill granules might also be used to match a granule of another product at a given time in a packaged file. Fill granules are not added at the beginning or end of an aggregation, and files that would consist entirely of fill granules are not produced. See section 3.5.6.1 of JPSS Common Data Format Control Book – External Volume I, p. 106 for more information.
á The beginning and end of each aggregation of any particular size for each NPP product is predetermined by integer numbers of aggregation of that size since t0 (1/1/1958). Aggregating a series of contiguous input granules the size of an aggregation or smaller may result in either one or two output files depending on whether the series falls within the predetermined bucket boundaries or straddles a bucket boundary. See JPSS Common Data Format Control Book – External Volume I, pp. 131-133 for more information.
For More Information
The nagg tool endeavors to produce files that match what the JPSS System provides as described in the Joint Polar Satellite System (JPSS) Common Data Format Control Book – External - Volumes I, III, IV, and V. These documents may be found at http://jointmission.gsfc.nasa.gov/science/documents.html.
Comments
Send questions or comments to the HDF Group Help Desk via help@hdfgroup.org.
The following are some notes about how to build, install, and run nagg.
A copy of the nagg source files can be downloaded from https://gamma.hdfgroup.org/ftp/pub/outgoing/JPSS/source/NAGG/.
nagg is a standalone tool that is run from the command line. The currently available version of the program can be built to work on 32- and 64-bit Linux systems.
The installation process is a file copy operation: copy the file to the directory where you want to keep the program.
In this version of the program, there is no file with configuration settings. To adjust the programÕs behavior, add one or more of the options to the command line statement when the program is executed. The available options are described in the ÒCommand SyntaxÓ section beginning on page 8.
Run the program from the directory where it is kept, or put in the path the directory where the program is stored.
Here are some notes that will help you build and test the nagg program.
Before you start:
á Make sure HDF5 version 1.8.5 or later is installed on your system. Source code or binaries can be downloaded from https://support.hdfgroup.org/HDF5/release/obtain5.html.
á Install the High-Level library for handling object and region references, hdf5_hl_region v.1.1.1; the source code can be found at https://gamma.hdfgroup.org/ftp/pub/outgoing/NPOESS/source/. For installation instructions see the README.txt file in the source distribution.
To build:
Assuming gmake and h5cc (HDF5 C compiler) are in your $PATH, just do
% ./configure --with-hlregion=/path-to-hdf5_hl_region-library
% gmake
The default C compiler is set to h5cc. If you want to use a specific h5cc compiler, do
% env CC=$mypath/h5cc ./configure
To build with different compiler flags, use the CFLAGS variable. An example is
% env CFLAGS=-O2 ./configure
This command will set the compiling to "h5cc -O2".
To run tests:
To run tests, use the following:
% gmake check
To install:
To install, use the following:
% gmake install
The nagg install files go to nagg/bin of the build directory by default. If you want to install it somewhere else, for example /usr/local/bin/nagg, use the following:
% gmake prefix=/usr/local install
The command line syntax for the nagg tool is described below. See the ÒCommand Line Options and ParametersÓ section on page 10 for more information.
Tool Name: nagg
Syntax
nagg –h | --version
nagg [--version] [-n number] [-t list | -l file] [-g criterion] [-S]
[-A
seconds] [-d directory] [-O origin] [-D domain] INPUTÉ
Purpose
To manipulate granules stored in NPP files into more easily processed NPP files.
Description
The command line utility nagg is the utility for grouping NPP granules into aggregate and package files.
Aggregates include time contiguous and gap-filled series of the same type of granules together in the same file. Packages include different, but related type granule aggregates with the same temporal and spatial extent together in the same file.
Input parameter INPUT is a list of one or more NPP files with the granules to aggregate and/or package.
The output of nagg is a file or set of files named according to the NPP file naming convention. Output files are aligned with the bucket boundary. For more information on bucket boundaries, see JPSS Common Data Format Control Book – External Volume I, pp. 131-133. Fill granules are generated when there are no data granules available in the input files for "slots" in a bucket. Leading and trailing fill granules are not generated for the first and last output files respectively. Therefore, first and last output files may contain fewer granules than requested.
The default behavior of nagg is to aggregate corresponding geolocation granules stored in an external geolocation file(s). The names of the geolocation file(s) can be found from the value(s) of an N_GEO_Ref attribute in the input file. The names of the output geolocation file(s) correspond to those of the primary product. nagg will fail if the input geolocation file is unavailable. The aggregating of geolocation granules can be overridden with the -g no command option which directs nagg to not use or look for geolocation input granules or files and to not produce geolocation output granules or files.
Geolocation file names can be approximate such that only the beginning parts up to the creation date are matched. Suppose the sensor data file defines its geolocation file as: GMTCO_npp_d20100906_t0701368_e0703013_b00004_c20111024161933653314_noaa_ops.h5. nagg will accept any file matching the following pattern and will use the one with the latest creation timestamp: GMTCO_npp_d20100906_t0701368_e0703013_b00004_c*.h5. The -g strict command option can be used to tell nagg that the external geolocation file name must exactly match the N_GEO_Ref value.
nagg is non-destructive. It will not overwrite existing files.
Tool Limits
Some limits for the nagg tool are defined below.
NAGG_Granule_info_max
10000 Maximum
number of granules managed
NAGG_Product_list_max
30 Maximum
number of products requested
NAGG_outputfiles_max
30 Maximum
number of output files
See ÒAppendix 2Ó in the NPP Aggregation Tool Components document for a complete list of size definitions.
Notes
The following are some notes about aggregating and de-aggregating files:
á nagg can de-aggregate files by setting the command line option -n to 1 and operating on aggregate files.
á nagg can aggregate files by setting the command line option -n to N and operating on a directory of single granule files.
á nagg
can re-aggregate files by setting the command line option -n
to a number different from the number of granules in the input files.
The command line options and parameters available with the nagg tool are described below. See the ÒCommand SyntaxÓ section above for more information.
-h, --help
Help. Using this command line option will display the command
syntax. This can also be used to list valid granule types and packaging
groupings.
--version
Displays the version of the nagg tool.
-t list, --type=list
Granule Type. list specifies a comma separated list of NPP record type mnemonics. Unless -S is specified, the granule types will be packaged together. Types must be compatible to be packaged together. Use -h to list valid package groupings. If –t is not specified, -g <product> must be given to aggregate only the geolocation product granules.
-n N, --number=N
Granule Number. N is the number of granules of each product in each aggregate file and must be greater than zero. If this option not specified, the default value of 1 (single granule files) will be used.
-g criterion, --geolocation=criterion
Geolocation Criterion. criterion is the criterion used for searching the geolocation granules. Valid values for criterion are the following:
|
no | 0 |
Aggregate product files without geolocation
input or output |
|
yes | 1 |
Allow approximate matching of geolocation
input file names (default) |
|
strict | 2 |
Require exact matching of geolocation
input file names |
|
<product> |
Aggregate geolocation product <product> only. This excludes
the use of –t. |
-l file
Example File. file is used by nagg as an example of
how to process data. Options
on the command line override the example file.
-A seconds
Aggregation Size. seconds is the number of seconds in each aggregate file. The aggregationÕs size is a fixed number of granules per file equal to seconds divided by the period of the granule rounded up to the next integer number of granules. The formula for this is
NumGranules/File = seconds/GranulePeriod
where
NumGranules/File is the number of granules in the aggregated file,
seconds is the number of seconds that are used to determine the size of the aggregated file, and
GranulePeriod is the number of seconds worth of data held by each granule.
If, for example, seconds is set to 60 and GranulePeriod is .7 seconds per granule, then the following steps show the calculation:
|
Calculation |
Comments |
|
NumGranules/File =
60 seconds/.7seconds/granule |
This is the setting of this parameter divided by the granule
period. |
|
NumGranules/File =
85.71 granules |
This is the actual value of the calculation. |
|
NumGranules/File =
86 granules |
This is the value of the calculation rounded up to the next
integer. |
If –l, -t, –n, and –A are omitted, then the first NPP data product file encountered will be used to determine the –t list and –n number.
This command line option has not yet been implemented.
-S, --simple
Simple Aggregates. Simple aggregates are produced. Each type is packaged separately. All granule types including geolocation products are packaged in one file.
-d directory, --directory=directory
Output Directory. Directory where the output files should be
placed. The default is the
current working directory. The tool will fail if the directory does not exist.
-O origin
Origin Identifier. The origin identifier is a four-character string in the output file name. Only the first four characters will be used. If fewer than four characters are specified, the origin will be pre-pended with the character ÒXÓ. The default is ÒXXXXÓ.
-D domain
Domain Identifier. The domain identifier is a three-character string in the output file name. Only the first three characters will be used. If fewer than three characters are specified, the domain will be pre-pended with the character ÒXÓ. The default is ÒXXXÓ.
--debug
Debug Mode. In debug mode, all of the granules in the input files including those not specified by the –t list are read in, all of the granules are displayed, and the execution is ended without generating the normal output files. Note that this option is intended for tool debugging. Files or output generated should not be used for production purposes: the behavior of debug mode may change from version to version.
INPUT
INPUT is
a list of one or more NPP files.
Exit Status
|
0 |
Succeeded. |
|
>0 |
An error occurred. |
The examples in this chapter demonstrate the use of nagg command line options in a variety of combinations. The input files for the examples are available in the test/testfiles directory of the nagg source files. The path in the commands will need modification depending on the directory chosen to run nagg.
Each example has a number of sub-sections. The sub-sections that are included vary by the action of the example. All of the examples have the following sub-sections: ÒCommand,Ó ÒCommand Line Elements,Ó ÒOutput,Ó ÒData Granules in Input and Output Files,Ó and ÒNotes.Ó The ÒCommandÓ sub-section shows the command line used in the example. The ÒCommand Line ElementsÓ sub-section describes how each element in the command line effects the results. The ÒOutputÓ sub-section describes the results of the command line. The ÒData Granules in Input and Output FilesÓ sub-section has a table that summarizes how the data granules are arranged in the input and output files. The ÒNotesÓ sub-section has items worth paying attention to.
The examples might have other sub-sections that further describe the input files, the output files, or any status messages produced by nagg during processing.
The contents of the input and output files can be viewed with the HDF5 utility h5ls. h5ls can be found in the bin directory of the HDF5 install. The command Òh5ls -r <filename>Ó recursively displays the full group path to every object in the file. For dataset objects, the name is followed by the dataset dimensions in the form {<current dimension>/<maximum dimension>, É}. See the ÒDe-aggregating an Aggregate FileÓ section on page 18 for an example. To see the differences described for the other examples, run Òh5ls –r <filename>Ó or use HDFView or some other utility to examine the file contents.
A granule in the ÒNotesÓ sub-sections will often refer to both the sensor data granule and the geolocation granule. See the example in the ÒAggregating Single Granule FilesÓ section on page 15: when it says that the first file has only one granule, the statement refers to both the first SATMS product file and also to the first GATMO product file with the corresponding geolocation data.
The table below lists the examples in this chapter.
|
Example |
See this page: |
|
Aggregating single granule files |
15 |
|
De-aggregating an aggregate file |
18 |
|
Re-aggregating by number of data
granules |
22 |
|
Re-aggregating by number of seconds of
data |
24 |
|
Packaging compatible products from
single granule input files |
26 |
|
Packaging compatible products from
single granule input files and fill granule files |
28 |
|
Packaging compatible products from
single granule input files using exact matching of geolocation
input file names |
32 |
|
Unpackaging and de-aggregating packaged and aggregated
input files |
33 |
|
Extracting a product from packaged
files containing other products |
36 |
|
Aggregating or de-aggregating geolocation files |
38 |
|
Re-aggregating granules from input
files like an example file |
39 |
|
-with -n command line override |
41 |
|
-with -A command line override |
43 |
|
-with -t command line override |
44 |
This example shows how to use nagg to aggregate single granule files.
Command
nagg –n 2 -t SATMS –S test/testfiles/SATMS_npp_d20100906*.h5
Command Line Elements
á The nagg command with -n 2 produces aggregations of two granules each.
á -t SATMS directs nagg to process SATMS product granules.
á nagg with no -g option specified processes SATMS product granules and by default the corresponding geolocation product granules either from the input files specified for the SATMS product or from files matching the file name in the N_GEO_Ref attribute in those files. For the default behavior, the creation date, origin, and domain description fields in the geolocation file name are replaced with * for matching files with the corresponding geolocation products that have the same date, begin time, end time, and orbit number (GATMO_npp_d20100906_t0750210_e0750524_b00005_*_ noaa_ops.h5).
á The -S option puts all products including the geolocation granules in separate files.
Output
The output of this command is three full files with two SATMS product granules each, three full files with two GATMO geolocation product granules each, one partial file with one SATMS granule, and one partial file with one GATMO granule.
Data Granules in Input and Output Files
|
Aggregations |
Total
granules |
Granules
per aggregation for each product |
Files
(containing full and partial aggregations) |
Hyperslabs in each raw data dataset for a full
aggregation |
|
Input aggregations |
14 |
1 |
14 |
1 |
|
Output aggregations |
14 |
2 |
8 (6 full, 2 partial) |
2 |
Notes
Why does the first file have only one granule when the command line option dictated two granules per file? nagg duplicates the behavior of the JPSS DDS: it calculates predetermined bucket boundaries according to integer multiples of the aggregation size beginning at t0 (12:00 am, 1/1/1958). For a more detailed explanation, see the ÒNotesÓ section on page 5 of this document or section 3.5.12 of the JPSS Common Data Format Control Book - External, Volume I, page 129. Since the first granule in these input files is in the second of two granules in a bucket, and since nagg does not produce leading or trailing fill granules, the first file has only the one granule. See the ÒPackaging Compatible Products from Single Granule Input Files and Fill Granule FilesÓ section on page 28 for more information.
The input file string test/testfiles/SATMS_npp_d20100906*.h5 matches seven
SATMS files in the test/testfiles directory, and
those seven files each have an N_GEO_Ref attribute
with the name of a GATMO file in the same directory. Unless -g
no
is supplied in the command line, nagg will
aggregate the geolocation granules in those seven
files provided that they exist. If all of the geolocation
files are missing, nagg will fail.
However, the product files can be aggregated without geolocation
using the -g no option.
Occasionally the creation time field of the geolocation
file names will not match the strings in the product filesÕ N_GEO_Ref
attributes. nagg handles
this difficulty by using the last created geolocation
file that matches the rest of the string. Exact matches can be enforced with
the -g strict option.
Input and output files may be examined using h5ls
–r as shown on page 19. HDFview and h5dump are other HDF5 tools that may be useful
for examining the contents of HDF5 files.
The effect on dataset dimensions in this example is the reverse of that for the example in the Ò3.2. De-aggregating an Aggregate FileÓ section starting on page 18: granules in this example are being aggregated rather than de-aggregated. With –n 2, the datasets in the /All_Data group of the output files will have first dimensions that are twice the size of the first dimensions in the input files. This reflects the two hyperslabs from two granules in the complete aggregations.
Input Files (14)
The input files are listed below.
test/testfiles/SATMS_npp_d20100906_t0750210_e0750524_b00005_c20111024161819049592_noaa_ops.h5
test/testfiles/SATMS_npp_d20100906_t0750530_e0751244_b00005_c20111024161819049592_noaa_ops.h5
test/testfiles/SATMS_npp_d20100906_t0751250_e0751564_b00005_c20111024161819049592_noaa_ops.h5
test/testfiles/SATMS_npp_d20100906_t0751570_e0752284_b00005_c20111024161819049592_noaa_ops.h5
test/testfiles/SATMS_npp_d20100906_t0752290_e0753004_b00005_c20111024161819049592_noaa_ops.h5
test/testfiles/SATMS_npp_d20100906_t0753010_e0753324_b00005_c20111024161819049592_noaa_ops.h5
test/testfiles/SATMS_npp_d20100906_t0753330_e0754044_b00005_c20111024161819049592_noaa_ops.h5
test/testfiles/GATMO_npp_d20100906_t0750210_e0750524_b00005_c20111024161819049592_noaa_ops.h5
test/testfiles/GATMO_npp_d20100906_t0750530_e0751244_b00005_c20111024161819049592_noaa_ops.h5
test/testfiles/GATMO_npp_d20100906_t0751250_e0751564_b00005_c20111024161819049592_noaa_ops.h5
test/testfiles/GATMO_npp_d20100906_t0751570_e0752284_b00005_c20111024161819049592_noaa_ops.h5
test/testfiles/GATMO_npp_d20100906_t0752290_e0753004_b00005_c20111024161819049592_noaa_ops.h5
test/testfiles/GATMO_npp_d20100906_t0753010_e0753324_b00005_c20111024161819049592_noaa_ops.h5
test/testfiles/GATMO_npp_d20100906_t0753330_e0754044_b00005_c20111024161819049592_noaa_ops.h5
Command Runtime Output
The status messages produced by the program during processing are listed below.
Produced 1 granules in SATMS_npp_d20100906_t0750210_e0750524_b00005_c20120516210214226724_XXXX_XXX.h5
Produced 1 granules in GATMO_npp_d20100906_t0750210_e0750524_b00005_c20120516210214226724_XXXX_XXX.h5
Produced 2 granules in SATMS_npp_d20100906_t0750530_e0751564_b00005_c20120516210214283013_XXXX_XXX.h5
Produced 2 granules in GATMO_npp_d20100906_t0750530_e0751564_b00005_c20120516210214283013_XXXX_XXX.h5
Produced 2 granules in SATMS_npp_d20100906_t0751570_e0753004_b00005_c20120516210214381909_XXXX_XXX.h5
Produced 2 granules in GATMO_npp_d20100906_t0751570_e0753004_b00005_c20120516210214381909_XXXX_XXX.h5
Produced 2 granules in SATMS_npp_d20100906_t0753010_e0754044_b00005_c20120516210214481149_XXXX_XXX.h5
Produced 2 granules in GATMO_npp_d20100906_t0753010_e0754044_b00005_c20120516210214481149_XXXX_XXX.h5
This example shows how to use nagg to de-aggregate an aggregate file.
Command
nagg -t SOMPS –S –O unkn –D dev test/testfiles/SOMPS_npp_d20120508_t0333549
_e0336247_b02735_c201205
22134023064953_unkn_xxx.h5
Command Line Elements
á The nagg command without -n <number> produces default aggregations of one granule each.
á -t SOMPS directs nagg to process SOMPS product granules.
á nagg with no -g option specified processes SOMPS product granules and by default the corresponding geolocation product granules either from the input files specified for the SOMPS product or from files matching the file name in the N_GEO_Ref attribute in those files. For the default behavior, the creation field in the geolocation file name is replaced with * for matching (GONPO_npp_d20120508_t0333549_e0336247_b02735_*_unkn_dev.h5).
á The -S option puts all products including the geolocation granules in separate files.
á -O unkn sets the origin field of the output file names to Òunkn.Ó The default if no –O option is provided is ÒXXXX.Ó
á -D dev sets the domain description field of the output file names to Òdev.Ó The default without the -D option is ÒXXX.Ó
Output
The output of this command is four files with one SOMPS product granule each and four files with one GONPO geolocation product granule each. The part of the output file names beginning with _c is determined by the creation time of the output file.
Data Granules in Input and Output Files
|
Aggregations |
Total
granules |
Granules
per aggregation for each product |
Files |
Hyperslabs in each raw data dataset for a full
aggregation |
|
Input aggregations |
8 |
4 |
2 |
4 |
|
Output aggregations |
8 |
1 |
8 |
1 |
Notes
The SOMPS input file has four SOMPS granules (see the ÒInput File ObjectsÓ section below) and a reference to the geolocation file which contains the corresponding geolocation granules. In this case, the four granules in the SOMPS file specified as input for the command plus four granules in the GONPO geolocation file it references result in eight output files: four with one SOMPS granule each and four with the corresponding GONPO granules.
The contents of the input and one output files as displayed by h5ls –r are shown below. The input file has four granules (highlighted in blue and bold), the output file shown has one granule. The other SOMPS output files have the same structure, each with one granule but with different data, and the four GONPO geolocation granules in the GONPO input file are similarly distributed into four output files with a different collection of datasets for the geolocation data. The datasets in the input files have four hyperslabs each, one for each granule. The datasets in the output files each have one of these hyperslabs for the fileÕs one granule. Consequently, the current dimensions (highlighted in yellow and underlined) for the first dimension of the datasets in the output files are one quarter the size of the same dimension in the input files (also highlighted in yellow and underlined). The time range from t0333549 to e0336247 reflected in the input file names is distributed among the four output files for each product.
Input File Objects
The listing of input file objects with the h5ls -r SOMPS_npp_d20120508_t0333549_e0336247_b02735_c20120522134023064953_unkn_xxx.h5 command line is shown below.
/ Group
/All_Data Group
/All_Data/OMPS-NP-SDR_All Group
/All_Data/OMPS-NP-SDR_All/BadCal Dataset {4/Inf}
/All_Data/OMPS-NP-SDR_All/Bias1 Dataset {4/Inf}
/All_Data/OMPS-NP-SDR_All/Cal Dataset {20/Inf, 200/Inf}
/All_Data/OMPS-NP-SDR_All/DarkCurrentEarth Dataset {24/Inf, 200/Inf}
É
/Data_Products Group
/Data_Products/OMPS-NP-SDR Group
/Data_Products/OMPS-NP-SDR/OMPS-NP-SDR_Aggr Dataset {21}
/Data_Products/OMPS-NP-SDR/OMPS-NP-SDR_Gran_0 Dataset {21/Inf}
/Data_Products/OMPS-NP-SDR/OMPS-NP-SDR_Gran_1 Dataset {21/Inf}
/Data_Products/OMPS-NP-SDR/OMPS-NP-SDR_Gran_2 Dataset {21/Inf}
/Data_Products/OMPS-NP-SDR/OMPS-NP-SDR_Gran_3 Dataset {21/Inf}
Output File Objects
The listing of objects in 1 of 8 output files with the h5ls –r SOMPS_npp_d20120508_t0333549_e0334324_b02735_c20120509170435469674_XXXX_XXX.h5 command line is shown below.
/ Group
/All_Data Group
/All_Data/OMPS-NP-SDR_All Group
/All_Data/OMPS-NP-SDR_All/BadCal Dataset {1/Inf}
/All_Data/OMPS-NP-SDR_All/Bias1 Dataset {1/Inf}
/All_Data/OMPS-NP-SDR_All/Cal Dataset {5/Inf, 200/Inf}
/All_Data/OMPS-NP-SDR_All/DarkCurrentEarth Dataset {6/Inf, 200/Inf}
É
/Data_Products Group
/Data_Products/OMPS-NP-SDR Group
/Data_Products/OMPS-NP-SDR/OMPS-NP-SDR_Aggr Dataset {21}
/Data_Products/OMPS-NP-SDR/OMPS-NP-SDR_Gran_0 Dataset {21/Inf}
Input Data File
The input data file is listed below.
SOMPS_npp_d20120508_t0333549_e0336247_b02735_c201205 22134023064953_unkn_xxx.h5
Input Geolocation File
The input geolocation data file is listed below.
GONPO_npp_d20120508_t0333549_e0336247_b02735_c201205 22134023064953_unkn_dev.h5
Command Runtime Output
The status messages produced by the program during processing are listed below.
Produced 1 granules in SOMPS_npp_d20120508_t0333549_e0334324_b02735_c20120509174130237345_unkn_dev.h5
Produced 1 granules in GONPO_npp_d20120508_t0333549_e0334324_b02735_c20120509174130237345_unkn_dev.h5
Produced 1 granules in SOMPS_npp_d20120508_t0334324_e0335098_b02735_c20120509174130259000_unkn_dev.h5
Produced 1 granules in GONPO_npp_d20120508_t0334324_e0335098_b02735_c20120509174130259000_unkn_dev.h5
Produced 1 granules in SOMPS_npp_d20120508_t0335098_e0335472_b02735_c20120509174130278264_unkn_dev.h5
Produced 1 granules in GONPO_npp_d20120508_t0335098_e0335472_b02735_c20120509174130278264_unkn_dev.h5
Produced 1 granules in SOMPS_npp_d20120508_t0335473_e0336247_b02735_c20120509174130297695_unkn_dev.h5
Produced 1 granules in GONPO_npp_d20120508_t0335473_e0336247_b02735_c20120509174130297695_unkn_dev.h5
This example shows how to use nagg to re-aggregate files by changing the number of data granules in each aggregated file from the original aggregated file.
Command
nagg –n 3 -t SATMS –S SATMS_npp_d20100906*_XXXX_XXX.h5
Command Line Elements
á The nagg command with -n 3 produces aggregations of three granules each.
á -t SATMS directs nagg to process SATMS product granules.
á nagg without a -g option behaves the same as Ò-g yesÓ and processes the GATMO geolocation product granules corresponding to the SATMS product either from the input files specified for the SATMS product or from files matching the file name in the N_GEO_Ref attribute in those files. For the default behavior, the creation field in the geolocation file name is replaced with * for matching (GATMO_npp_d20100906_t0750210_e0750524_b00005_*_ XXXX_XXX.h5).
á The -S option puts all products including the geolocation granules in separate files.
Output
The output of this command is two full files with three SATMS product granules each, two full files with three GATMO geolocation product granules each, one partial file with one SATMS granule, and one partial file with one GATMO granule.
Data Granules in Input and Output Files
|
Aggregations |
Total
granules |
Granules
per aggregation for each product |
Files
(containing full and partial aggregations) |
Hyperslabs in each raw data dataset for a full
aggregation |
|
Input aggregations |
14 |
2 |
8 (6 full, 2 partial) |
2 |
|
Output aggregations |
14 |
3 |
6 (4 full, 2 partial) |
3 |
Notes
This example uses the output files from the example in the ÒAggregating Single Granule FilesÓ section on page 15. As it happens, the granule bucket boundaries for two and three granule aggregations coincide between the first and second granules in this set of granules, so the first file once again has only the last granule since nagg does not produce leading fill granules. For this set of granules, the first granule is the only granule of a partial aggregation.
Input Files (8)
The input files are listed below.
SATMS_npp_d20100906_t0750210_e0750524_b00005_c20120516210214226724_XXXX_XXX.h5
SATMS_npp_d20100906_t0750530_e0751564_b00005_c20120516210214283013_XXXX_XXX.h5
SATMS_npp_d20100906_t0751570_e0753004_b00005_c20120516210214381909_XXXX_XXX.h5
SATMS_npp_d20100906_t0753010_e0754044_b00005_c20120516210214481149_XXXX_XXX.h5
GATMO_npp_d20100906_t0750210_e0750524_b00005_c20120516210214226724_XXXX_XXX.h5
GATMO_npp_d20100906_t0750530_e0751564_b00005_c20120516210214283013_XXXX_XXX.h5
GATMO_npp_d20100906_t0751570_e0753004_b00005_c20120516210214381909_XXXX_XXX.h5
GATMO_npp_d20100906_t0753010_e0754044_b00005_c20120516210214481149_XXXX_XXX.h5
Command Runtime Output
The status messages produced by the program during processing are listed below.
Produced 1 granules in SATMS_npp_d20100906_t0750210_e0750524_b00005_c20120516210313080299_XXXX_XXX.h5
Produced 1 granules in GATMO_npp_d20100906_t0750210_e0750524_b00005_c20120516210313080299_XXXX_XXX.h5
Produced 3 granules in SATMS_npp_d20100906_t0750530_e0752284_b00005_c20120516210313138410_XXXX_XXX.h5
Produced 3 granules in GATMO_npp_d20100906_t0750530_e0752284_b00005_c20120516210313138410_XXXX_XXX.h5
Produced 3 granules in SATMS_npp_d20100906_t0752290_e0754044_b00005_c20120516210313277669_XXXX_XXX.h5
Produced 3 granules in GATMO_npp_d20100906_t0752290_e0754044_b00005_c20120516210313277669_XXXX_XXX.h5
This example shows how to use nagg to re-aggregate data by number of seconds of data rather than by number of data granules.
Command
nagg –A 300 –t REDRO –d npp-out test/testfiles/REDRO_npp_d20030125*.h5
Command Line Elements
á The nagg command with –A 300 for the REDRO product produces aggregations of ten granules each. This is determined by dividing 300 by the 31.997 second granule duration and rounding up to the next integer (10).
á -t REDRO directs nagg to process REDRO product granules.
á nagg without a -g option behaves the same as Ò-g yesÓ and processes the GCRIO geolocation product granules corresponding to the REDRO product either from the input files specified for the REDRO product or from files matching the file name in the N_GEO_Ref attribute in those files. For the default behavior, the creation field in the geolocation file name is replaced with * for matching (GCRIO_npp_ d20030125_t0702533_e0711257_b00014_*_unkn_ada.h5).
á Without the -S option the SDR data granules and the corresponding geolocation granules are put in one output file for each aggregation.
á With the –d <directory> command line option, the output files are created in the specified npp-out directory.
Output
The output of this command is six full files with ten REDRO product granules and ten GCRIO geolocation granules in each file and one partial file with four REDRO product granules and four GCRIO geolocation product granules.
Data Granules in Input and Output Files
|
Aggregations |
Total
granules |
Granules
per aggregation for each product |
Files
(containing full and partial aggregations) |
Hyperslabs in each raw data dataset for a full
aggregation |
|
Input aggregations |
128 |
16 |
8 |
16 |
|
Output aggregations |
128 |
10 |
7 (6 full, 1 partial) |
10 |
Notes
For this set of files, the first granule in the first input file happens to be the first granule in a bucket for a 16 granule aggregation and also for a 10 granule aggregation. The last file is a partial aggregation with only four granules.
Either relative or absolute paths are allowed for the directory specified by –d.
Input Files (8)
The input files are listed below.
test/testfiles/REDRO_npp_d20030125_t0702533_e0711257_b00014_c20111025170507190780_unkn_ada.h5
test/testfiles/REDRO_npp_d20030125_t0711279_e0719577_b00014_c20111025170507195067_unkn_ada.h5
test/testfiles/REDRO_npp_d20030125_t0719599_e0728297_b00014_c20111025170507197969_unkn_ada.h5
test/testfiles/REDRO_npp_d20030125_t0728319_e0737011_b00014_c20111025170507204040_unkn_ada.h5
test/testfiles/GCRIO_npp_d20030125_t0702533_e0711257_b00014_c20111025170507184515_unkn_ada.h5
test/testfiles/GCRIO_npp_d20030125_t0711279_e0719577_b00014_c20111025170507194756_unkn_ada.h5
test/testfiles/GCRIO_npp_d20030125_t0719599_e0728297_b00014_c20111025170507197657_unkn_ada.h5
test/testfiles/GCRIO_npp_d20030125_t0728319_e0737011_b00014_c20111025170507201941_unkn_ada.h5
Command Runtime Output
The status messages produced by the program during processing are listed below.
Produced 10 granules in npp-out/GCRIO-REDRO _npp_d20030125_t0702533_e0708137_b00993_c20120712162901513435_XXXX_XXX.h5
Produced 10 granules in npp-out/GCRIO-REDRO_ npp_d20030125_t0708159_e0713337_b00014_c20120712162901631182_XXXX_XXX.h5
Produced 10 granules in npp-out/GCRIO-REDRO _npp_d20030125_t0713359_e0718537_b00014_c20120712162901795401_XXXX_XXX.h5
Produced 10 granules in npp-out/GCRIO-REDRO_ npp_d20030125_t0718559_e0724137_b00014_c20120712162901920678_XXXX_XXX.h5
Produced 10 granules in npp-out/GCRIO-REDRO _npp_d20030125_t0724159_e0729337_b00014_c20120712162902111364_XXXX_XXX.h5
Produced 10 granules in npp-out/GCRIO-REDRO _npp_d20030125_t0729359_e0734531_b00014_c20120712162902271830_XXXX_XXX.h5
Produced 4 granules in npp-out/GCRIO-REDRO _npp_d20030125_t0734531_e0737011_b00993_c20120712162902398737_XXXX_XXX.h5
This example shows how to use nagg
to package compatible products from single granule
input files.
Command
nagg –n 5 -t SATMS,TATMS test/testfiles/SATMS_npp_d2012040[34]*.h5 test/testfiles/TATMS_npp_d2012040[34]*.h5
Command Line Elements
á The nagg command with -n 5 produces aggregations of five granules each.
á -t SATMS,TATMS indicates to nagg that those two products and their common geolocation granules are to be aggregated.
á nagg with no -g option processes granules for the products specified with -t and by default for the corresponding geolocation product granules either from the input files specified for the -t products or from files matching the file name in the N_GEO_Ref attribute in those files. For the default behavior, the creation field in the geolocation file name is replaced with * for matching (GONPO_npp_d20120508_t0333549_e0336247_b02735_*_unkn_dev.h5).
á Without the -S option, all products including the geolocation granules are put in a single file for each aggregation.
Output
The output of this command is one partial file with four SATMS and four TATMS product granules and four GATMO geolocation product granules and one other partial file with one granule of each product.
Data Granules in Input and Output Files
|
Aggregations |
Total
granules |
Granules
per aggregation for each product |
Files
(containing full and partial aggregations) |
Hyperslabs in each raw data dataset for a full
aggregation |
|
Input aggregations |
15 |
1 |
15 |
1 |
|
Output aggregations |
15 |
5 |
2 partial |
5 |
Notes
The [34]* wildcards in the –t option in the command line matches all SATMS_npp_d20120403É and SATMS_npp_d20120404É file names in the input directory; the wildcards also match those with the same file name patterns beginning with TATMS. The corresponding GATMO files are included based on the /N_GEO_Ref attributes in the files for both products. For packaged files with multiple products, only one corresponding geolocation granule will be included in the output: all packaged products must have the same corresponding geolocation granules, and duplicate granules are added only once.
The command asked for a five granule aggregation. In this case, five input granules for each product were supplied, but a predetermined aggregation boundary for five granules of these products falls after the first four input granules. Since leading and trailing fill granules are not produced, the aggregation produces two files: the first with four sets of granules and the second with one set of granules.
Examining the contents of the input and output files will show that the first dimension of the datasets in the /All_Data group of the output file are five times the size of the first dimension of the datasets with the same name in the input files. The output files raw data datasets contain hyperslabs for each of five granules compared to one hyperslab in the input filesÕ datasets.
Input Files (15)
The input files are listed below.
test/testfiles/SATMS_npp_d20120403_t2359123_e2359439_b02250_c20120405155447242391_noaa_ops.h5
É
test/testfiles/SATMS_npp_d20120404_t0001203_e0001519_b02250_c20120405155447242391_noaa_ops.h5
test/testfiles/TATMS_npp_d20120403_t2359123_e2359439_b02250_c20120405155457899734_noaa_ops.h5
É
test/testfiles/TATMS_npp_d20120404_t0001203_e0001519_b02250_c20120405155457899734_noaa_ops.h5
test/testfiles/GATMO_npp_d20120403_t2359123_e2359439_b02250_c20120405155441643487_noaa_ops.h5
É
test/testfiles/GATMO_npp_d20120404_t0001203_e0001519_b02250_c20120405155441643487_noaa_ops.h5
Command Runtime Output
The status messages produced by the program during processing are listed below.
Produced 4 granules in SATMS-TATMS-GATMO_npp_d20120403_t2359123_e0001199_b02250_c20120510175020763627_XXXX_XXX.h5
Produced 1 granules in SATMS-TATMS-GATMO_npp_d20120404_t0001203_e0001519_b02250_c20120510175020886072_XXXX_XXX.h5
This example shows how to use nagg
to package compatible products from single granule
input files and shows the use of fill granule files.
Command
nagg –n 5 -t SATMS,TATMS test/testfiles/SATMS_npp_d2012040[34]*.h5 test/testfiles/TATMS_npp_d20120404*.h5
Command Line Elements
á The nagg command with -n 5 produces aggregations of five granules each.
á -t SATMS,TATMS indicates to nagg that those two products and their common geolocation data are to be aggregated.
á nagg with no -g option processes granules for the products specified with -t and by default for the corresponding geolocation product granules either from the input files specified for the -t products or from files matching the file name in the N_GEO_Ref attribute in those files. For the default behavior, the creation field in the geolocation file name is replaced with * for matching (GONPO_npp_d20120508_t0333549_e0336247_b02735_*_unkn_dev.h5).
á Without the -S option, all products including the geolocation granules are put in a single file for each aggregation.
Output
The output of this command is one partial file with four SATMS and four TATMS product granules and four GATMO geolocation product granules and one other partial file with one granule of each product.
Data Granules in Input and Output Files
|
Aggregations |
Total
granules |
Granules
per aggregation for each product |
Files
(containing full and partial aggregations) |
Hyperslabs in each raw data dataset for a full
aggregation |
|
Input aggregations |
13 |
1 |
15 |
1 |
|
Output aggregations |
15 |
5 |
2 partial |
5 |
Notes
The [34]* wildcards in the –t option in the command line matches all SATMS_npp_d20120403É and SATMS_npp_d20120404É file names in the input directory. Those beginning with TATMS match only TATMS_ npp_d20120404É file names. The corresponding GATMO files are included as they are in the previous example (see page 26), and the aggregation boundaries, numbers of granules in output files, and dimensions of raw data datasets are also the same.
The difference in output from the previous example is that with five input SATMS and GATMO granules but only three input TATMS granules, nagg produces two TATMS fill granules to complete the packaging. nagg will also produce fill granules for granules missing from between granules in a temporal aggregation.
These fill granules can be identified by examining the output files to check the N_Granule_ID attributes of the granule datasets with HDF5 tools HDFview or h5dump as shown below. h5dump is another HDF5 utility found in the bin directory of the HDF5 install. Using h5dump to view the N_Granule_ID of the first SATMS granule and the first TATMS granule shows the difference in N_Granule_IDs for a real and a fill granule. The difference is the real granule ID has a 15 character string beginning with ÒNPPÓ while the fill granule ID is ÒN/AÓ. Fill granules will have the same amount of data as real granules, but the data will be fill values as specified in the JPSS Common Data Format Control Book.
Real Granule Command
The following h5dump command line will retrieve a real granule ID.
h5dump -a /Data_Products/ATMS-SDR/ATMS-SDR_Gran_0/N_Granule_ID SATMS-TATMS-GATMO_npp_d20120403_t2359123_e0001199_b02250_c20120529123052142977_XXXX_XXX.h5
Real Granule Output
The output from the real granule command is below.
HDF5 "SATMS-TATMS-GATMO_npp_d20120403_t2359123_e0001199_b02250_c20120529123052142977_XXXX_XXX.h5" {
ATTRIBUTE "/Data_Products/ATMS-SDR/ATMS-SDR_Gran_0/N_Granule_ID" {
É
DATA {
(0,0): "NPP000141695514"
}
}
Fill Granule Command
The following h5dump command line will retrieve a fill granule ID.
h5dump -a /Data_Products/ATMS-TDR/ATMS-TDR_Gran_0/N_Granule_ID SATMS-TATMS-GATMO_npp_d20120403_t2359123_e0001199_b02250_c20120529123052142977_XXXX_XXX.h5
Fill Granule Output
The output from the fill granule command is below.
HDF5 "SATMS-TATMS-GATMO_npp_d20120403_t2359123_e0001199_b02250_c20120529123052142977_XXXX_XXX.h5" {
ATTRIBUTE "/Data_Products/ATMS-TDR/ATMS-TDR_Gran_0/N_Granule_ID" {
É
DATA {
(0,0): "N/A"
}
}
}
Input Files (13)
The input files are listed below.
test/testfiles/SATMS_npp_d20120403_t2359123_e2359439_b02250_c20120405155447242391_noaa_ops.h5
É
test/testfiles/SATMS_npp_d20120404_t0001203_e0001519_b02250_c20120405155447242391_noaa_ops.h5
test/testfiles/TATMS_npp_d20120404_t0000163_e0000479_b02250_c20120405155457899734_noaa_ops.h5
É
test/testfiles/TATMS_npp_d20120404_t0001203_e0001519_b02250_c20120405155457899734_noaa_ops.h5
test/testfiles/GATMO_npp_d20120403_t2359123_e2359439_b02250_c20120405155441643487_noaa_ops.h5
É
test/testfiles/GATMO_npp_d20120404_t0001203_e0001519_b02250_c20120405155441643487_noaa_ops.h5
Command Runtime Output
The status messages produced by the program during processing are listed below.
Produced 4 granules in SATMS-TATMS-GATMO_npp_d20120403_t2359123_e0001199_b02250_c20120529123052142977_XXXX_XXX.h5
Produced 1 granules in SATMS-TATMS-GATMO_npp_d20120404_t0001203_e0001519_b02250_c20120529123052282010_XXXX_XXX.h5
This example shows how to use nagg
to package compatible products from single granule
input files and how to specify the exact matching of geolocation
input file names.
Command
nagg –n 5 -t SATMS,TATMS –g strict test/testfiles/SATMS_npp_d2012040[34]*.h5 test/testfiles/TATMS_npp_d2012040[34]*.h5
This is the same command and same set of input files as the example
in the ÒPackaging Compatible Products from Single Granule Input FilesÓ on page 26 except
for the –g strict command line option. Adding –g
strict to the command will enforce that geolocation
file names must exactly match those in the /N_GEO_Ref
attribute of the SATMS and TATMS files. Since the name of the GATMO file with
the geolocation granule corresponding to the first
SATMS or TATMS granule processed, GATMO_npp_d20120403_t2359123_e2359439_b02250_c20120405155441643487_noaa_ops.h5, has a
creation time that does not match that in the attribute, the command fails. With
-g yes (the default) instead of -g
strict, the creation time, origin, and domain fields are masked when
searching for the geolocation file named in the
attribute. The -g no option processes the data files without
processing the geolocation information.
Command Runtime Output
The status messages produced by the program during processing are listed below.
nagg: ***ERROR*** find_geo_granules(): Failed to get geolocation granules.
test/testfiles/GATMO_npp_d20120403_t2359123_e2359439_b02250_c20120405155447242391_noaa_ops.h5 does not exist or otherwise cannot be accessed for reading.
nagg: ***ERROR*** nagg_get_granules(): Failed to get geolocation granules
nagg: ***ERROR*** nagg_get_granules(): no granules found for geoproduct.
geoproduct: GATMO
nagg: ***ERROR*** Failure encountered in Get Granules.
This example shows how to use nagg to unpackage and de-aggregate packaged and aggregated input files.
Command
nagg -t SVM03,SVM04,SVM05 -S -g no test/testfiles/SVM03-SVM04-SVM05_npp_d20120424_t0010313_e0016117_b02534_c20120521190943233596_unkn_xxx.h5
Command Line Elements
á The nagg command without the -n <number> command line option produces default aggregations of one granule each.
á The -S option puts all product granules in separate files.
á
-t SVM03,SVM04,SVM05 indicates
to nagg that granules of those products are
to be de-aggregated and unpackaged.
á
The -g no option
tells nagg to ignore the corresponding geolocation granules, neither trying to find them for input
nor to produce geolocation information in output files.
This was done for this example because the GMODO geolocation
files for VIIRS products are rather large for distributing with the source code.
The -g no option may also be useful for using nagg with product files when the geolocation files are unavailable.
Output
The output of this command is 12 files: four for each of the three products.
Data Granules in Input and Output Files
|
Aggregations |
Total
granules |
Granules
per aggregation for each product |
Files |
Hyperslabs in each raw data dataset for a full
aggregation |
|
Input aggregations |
12 |
4 |
1 |
4 |
|
Output aggregations |
12 |
1 |
12 |
1 |
Notes
Examining the contents of the input and output files will show that the first dimension of the datasets in the All_Data group of the output file are one quarter the size of the first dimension of the corresponding dataset in the input files. The input fileÕs raw data datasets contain hyperslabs for each of four granules, and the output filesÕ datasets contain a hyperslab for only one granule.
Input File
The input file is listed below.
test/testfiles/SVM03-SVM04-SVM05_npp_d20120424_t0010313_e0016117_b02534_c20120521190943233596_unkn_xxx.h5
Command Runtime Output
The status messages produced by the program during processing are listed below.
Produced 1 granules in SVM03_npp_d20120424_t0010313_e0011555_b02534_c2 0120521192724535803_XXXX_XXX.h5
Produced 1 granules in SVM04_npp_d20120424_t0010313_e0011555_b02534_c20120521192724535803_XXXX_XXX.h5
Produced 1 granules in SVM05_npp_d20120424_t0010313_e0011555_b02534_c20120521192724535803_XXXX_XXX.h5
Produced 1 granules in SVM03_npp_d20120424_t0011567_e0013209_b02534_c20120521192724857446_XXXX_XXX.h5
Produced 1 granules in SVM04_npp_d20120424_t0011567_e0013209_b02534_c20120521192724857446_XXXX_XXX.h5
Produced 1 granules in SVM05_npp_d20120424_t0011567_e0013209_b02534_c20120521192724857446_XXXX_XXX.h5
Produced 1 granules in SVM03_npp_d20120424_t0013221_e0014463_b02534_c20120521192725180119_XXXX_XXX.h5
Produced 1 granules in SVM04_npp_d20120424_t0013221_e0014463_b02534_c20120521192725180119_XXXX_XXX.h5
Produced 1 granules in SVM05_npp_d20120424_t0013221_e0014463_b02534_c20120521192725180119_XXXX_XXX.h5
Produced 1 granules in SVM03_npp_d20120424_t0014475_e0016117_b02534_c20120521192725504013_XXXX_XXX.h5
Produced 1 granules in SVM04_npp_d20120424_t0014475_e0016117_b02534_c20120521192725504013_XXXX_XXX.h5
Produced 1
granules in SVM05_npp_d20120424_t0014475_e0016117_b0253 4_c20120521192725504013_XXXX_XXX.h5
This example shows how to use nagg
to extract a product from packaged files containing
other products.
Command
nagg –n 4 –t SVM04 -g no test/testfiles/SVM03-SVM04-SVM05_npp_d20120424_t0010313_e0016117_b02534_c20120521190943233596_unkn_xxx.h5
Command Line Elements
á The nagg command with the -n 4 command line option produces aggregations of four granules each.
á
-t SVM04 indicates
to nagg that only granules of the SVM04 product
are to be aggregated and packaged in the output file.
á
The -g no option
tells nagg to ignore the corresponding geolocation granules, neither trying to find them for input
nor to produce geolocation information in output
files. This was done for this example because the GMODO geolocation
files for VIIRS products are rather large for distributing with the source
code. The -g no option may also be useful for using nagg with product files when the geolocation files are unavailable.
Output
The output of this command is one file with four granules of the SVM04 product only.
Data Granules in Input and Output Files
|
Aggregations |
Total
granules |
Granules
per aggregation for each product |
Files |
Hyperslabs in each raw data dataset for a full
aggregation |
|
Input aggregations |
12 |
4 |
1 |
4 |
|
Output aggregations |
4 |
4 |
1 |
4 |
Notes
This example uses the same input file as the example in the ÒUnpackaging and De-aggregating Packaged and Aggregated Input FilesÓ section on page 33, but only one of the products is listed for the –t option. The aggregation size (4) is the same as in the input file, so nagg in effect copies the listed product only into the output file.
For this example the granules per aggregation of each product, and consequently the number of hyperslabs in each dataset, are the same in the output file as in the input file. This time the total number of granules in the output file is smaller than the original because only one of three products is selected for aggregation in the output file.
Input File
The input file is listed below.
test/testfiles/SVM03-SVM04-SVM05_npp_d20120424_t0010313_e0016117_b02534_c20120521190943233596_unkn_xxx.h5
Command Runtime Output
The status message produced by the program during processing is listed below.
Produced 4 granules in SVM04_npp_d20120424_t0010313_e0016117_b02534_c20120521202951397965_XXXX_XXX.h5
This example shows how to use nagg to aggregate or de-aggregate geolocation files.
Command
nagg –n32 -g GCRIO ../test/testfiles/GCRIO_npp_d20030125*.h5
Command Line Elements
á The nagg command with the -n 32 command line option produces aggregations of 32 granules each.
á
The -g <product> option is
for aggregating or de-aggregating granules of a geolocation
product directly without any corresponding sensor date product granules. -t is not used with –g
<product>.
Output
The output of this command is three files with 16 or 32 granules of the GCRIO product only.
Data Granules in Input and Output Files
|
Aggregations |
Total
granules |
Granules
per aggregation for each product |
Files
(containing full and partial aggregations) |
Hyperslabs in each raw data dataset for a full
aggregation |
|
Input aggregations |
79 |
16 |
5 (4 full 1 partial) |
16 |
|
Output aggregations |
64 |
32 |
3 (1 full, 2 partial) |
32 |
Notes
The -g <product> command line option is provided for processing aggregations that contain only geolocation granules.
In this example the total number of granules in the output files is smaller than the total number of granule in the input files. The second input file with the name ending in anyt_ing.h5 contains duplicates of granules in the first and third input files. nagg selects only one of the duplicates for the output aggregation; the other 15 duplicates are ignored.
Input Files
The input files are listed below.
test/testfiles/GCRIO_npp_d20030125_t0702533_e0711257_b00014_c20111025170507184515_unkn_ada.h5
test/testfiles/GCRIO_npp_d20030125_t0705359_e0713337_b00014_c98765432109876543210_anyt_ing.h5
test/testfiles/GCRIO_npp_d20030125_t0711279_e0719577_b00014_c20111025170507194756_unkn_ada.h5
test/testfiles/GCRIO_npp_d20030125_t0719599_e0728297_b00014_c20111025170507197657_unkn_ada.h5
test/testfiles/GCRIO_npp_d20030125_t0728319_e0737011_b00014_c20111025170507201941_unkn_ada.h5
Command Runtime Output
The status messages produced by the program during processing are listed below.
Produced 16 granules in GCRIO_npp_d20030125_t0702533_e0711257_b00993_c20120523155444015673_XXXX_XXX.h5
Produced 32 granules in GCRIO_npp_d20030125_t0711279_e0728297_b00014_c20120523155445226530_XXXX_XXX.h5
Produced 16 granules in GCRIO_npp_d20030125_t0728319_e0737011_b00014_c20120523155447399983_XXXX_XXX.h5
This example shows how to use nagg –l or --like <example file> to aggregate or package granules from input files with a product list and aggregation number like those in the example file. Note that the aggregation number will be the number of granules per product in the file, and if the file is a partial file, the aggregation number will be less than the original aggregation number. Avoid beginning and ending files to insure the full number of granules per file in the original aggregation.
Command
nagg –l ../test/testfiles/GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0000159_b02250_c20120830155722719166_unkn_xxx.h5 ../test/testfiles/SATMS_npp_d2012040*.h5 ../test/testfiles/TATMS_npp_d2012040
Command Line Elements
á -l example file GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0000159_b02250_c20120830155722719166_unkn_xxx.
Output
The output of this command is three files with 2 or 1 granules of the SATMS, TATMS, and GATMO products.
Data Granules in Input and Output Files
|
Aggregations |
Total
granules |
Granules
per aggregation for each product |
Files
(containing full and partial aggregations) |
Hyperslabs in each raw data dataset for a full
aggregation |
|
Input aggregations |
15 |
1 |
15 |
1 |
|
Output aggregations |
15 |
2 |
3 (2 full, 1 partial) |
2 |
Notes
The example file GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0000159_b02250_c20120830155722719166_unkn_xxx.h5has 2 granules for each of these products: SATMS, TATMS and the GATMO GEO product. The nagg command with the –l GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0000159_b02250_c20120830155722719166_unkn_xxx.h5 command line option produces aggregations of 2 granules each with SATMS and TATMS granules found in the input files plus corresponding GEO granules.
The -l <example file> option determines the number of granules per output file and the list of sensor data products to be aggregated/packaged by inspection of the example file. All other properties are determined by other command line options or nagg defaults. Use of the -l option with geolocation only files is not supported. The -A, -n, or -t command options will override the corresponding value from the example file.
Input Files
The input files are listed below.
../test/testfiles/SATMS_npp_d20120403_t2359123_e2359439_b02250_c20120405155447242391_noaa_ops.h5
../test/testfiles/SATMS_npp_d20120403_t2359443_e0000159_b02250_c20120405155447242391_noaa_ops.h5
../test/testfiles/SATMS_npp_d20120404_t0000163_e0000479_b02250_c20120405155447242391_noaa_ops.h5
../test/testfiles/SATMS_npp_d20120404_t0000483_e0001199_b02250_c20120405155447242391_noaa_ops.h5
../test/testfiles/SATMS_npp_d20120404_t0001203_e0001519_b02250_c20120405155447242391_noaa_ops.h5
../test/testfiles/TATMS_npp_d20120403_t2359123_e2359439_b02250_c20120405155457899734_noaa_ops.h5
../test/testfiles/TATMS_npp_d20120403_t2359443_e0000159_b02250_c20120405155457899734_noaa_ops.h5
../test/testfiles/TATMS_npp_d20120404_t0000163_e0000479_b02250_c20120405155457899734_noaa_ops.h5
../test/testfiles/TATMS_npp_d20120404_t0000483_e0001199_b02250_c20120405155457899734_noaa_ops.h5
../test/testfiles/TATMS_npp_d20120404_t0001203_e0001519_b02250_c20120405155457899734_noaa_ops.h5
../test/testfiles/GATMO_npp_d20120403_t2359123_e2359439_b02250_c20120405155441643487_noaa_ops.h5
../test/testfiles/GATMO_npp_d20120403_t2359443_e0000159_b02250_c20120405155441643487_noaa_ops.h5
../test/testfiles/GATMO_npp_d20120404_t0000163_e0000479_b02250_c20120405155441643487_noaa_ops.h5
../test/testfiles/GATMO_npp_d20120404_t0000483_e0001199_b02250_c20120405155441643487_noaa_ops.h5
../test/testfiles/GATMO_npp_d20120404_t0001203_e0001519_b02250_c20120405155441643487_noaa_ops.h5
Command Runtime Output
The status messages produced by the program during processing are listed below.
Produced 2 granules in GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0000159_b02250_c20120831023912403398_XXXX_XXX.h5
Produced 2 granules in
GATMO-SATMS-TATMS_npp_d20120404_t0000163_e0001199_b02250_c20120831023912464828_XXXX_XXX.h5
Produced 1 granules in
GATMO-SATMS-TATMS_npp_d20120404_t0001203_e0001519_b02250_c20120831023912550539_XXXX_XXX.h5
The -l or –like option in the previous example can be used in combination with -n, -A, or -t. When one of these options is also in the command line, it will override the conflicting directive from the –like <example file> option. The following example uses the same example file and input files as in the previous example.
Command
nagg –n 3 --like ../test/testfiles/GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0000159_b02250_c20120830155722719166_unkn_xxx.h5 ../test/testfiles/SATMS_npp_d2012040*.h5 ../test/testfiles/TATMS_npp_d2012040
Command Line Elements
á -l example file GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0000159_b02250_c20120830155722719166_unkn_xxx.
á -n 3 overrides the aggregation number from the –l option.
Output
The output of this command is two files with 3 or 2 granules of the SATMS, TATMS, and GATMO products.
Data Granules in Input and Output Files
|
Aggregations |
Total
granules |
Granules
per aggregation for each product |
Files
(containing full and partial aggregations) |
Hyperslabs in each raw data dataset for a full
aggregation |
|
Input aggregations |
15 |
1 |
15 |
1 |
|
Output aggregations |
15 |
3 |
2 (1 full, 1 partial) |
3 |
Notes
á With the addition of –n 3 the nagg command with the –l GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0000159_b02250_c20120830155722719166_unkn_xxx.h5 command line option produces aggregations of 3 granules each with SATMS and TATMS granules found in the input files plus corresponding GEO granules.
Command Runtime Output
The status messages produced by the program during processing are listed below.
Produced 3 granules in
GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0000479_b02250_c20120831160042219489_XXXX_XXX.h5
Produced 2 granules in GATMO-SATMS-TATMS_npp_d20120404_t0000483_e0001519_b02250_c20120831160042440599_XXXX_XXX.h5
The -l or –like option in the previous example can be used in combination with -n, -A, or -t. When one of these options is also in the command line, it will override the conflicting directive from the –like <example file> option. The following examples use the same example file and input files as in the example above.
Suppose 2 minutes of SATMS, TATMS and associated GATMO data were desired instead of the 2 granules in the example file. –A 120 will provide an aggregation of that size, with the products according to the example file.
Command
nagg –A 120 --like ../test/testfiles/GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0000159_b02250_c20120830155722719166_unkn_xxx.h5 ../test/testfiles/SATMS_npp_d2012040*.h5 ../test/testfiles/TATMS_npp_d2012040
Command Line Elements
á -l example file GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0000159_b02250_c20120830155722719166_unkn_xxx.
á -A 120 overrides the aggregation number from the –l option.
Output
The output of this command is two files with 4 or 1 granules of the SATMS, TATMS, and GATMO products.
Data Granules in Input and Output Files
|
Aggregations |
Total
granules |
Granules
per aggregation for each product |
Files
(containing full and partial aggregations) |
Hyperslabs in each raw data dataset for a full
aggregation |
|
Input aggregations |
15 |
1 |
15 |
1 |
|
Output aggregations |
15 |
4 |
2 (1 full, 1 partial) |
4 |
Notes
á With the addition of –n 120 the nagg command with the –l GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0000159_b02250_c20120830155722719166_unkn_xxx.h5 command line option produces aggregations of 4 granules each with SATMS and TATMS granules found in the input files plus corresponding GEO granules.
Command Runtime Output
The status messages produced by the program during processing are listed below.
Produced 4 granules in
GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0001199_b02250_c20120831161244023811_XXXX_XXX.h5
Produced 1 granules in
GATMO-SATMS-TATMS_npp_d20120404_t0001203_e0001519_b02250_c20120831161244313132_XXXX_XXX.h5
The -l or –like option in the previous example can be used in combination with -n, -A, or -t. When one of these options is also in the command line, it will override the conflicting directive from the –like <example file> option. The following examples use the same example file and input files as in the example above.
Overriding the -l option with a –t <product list> option will use the aggregation number from the file, but the –t product list.
Command
nagg –t TATMS --like ../test/testfiles/GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0000159_b02250_c20120830155722719166_unkn_xxx.h5 ../test/testfiles/SATMS_npp_d2012040*.h5 ../test/testfiles/TATMS_npp_d2012040
Command Line Elements
á -l example file GATMO-SATMS-TATMS_npp_d20120403_t2359123_e0000159_b02250_c20120830155722719166_unkn_xxx.
á -t TATMS overrides the product list from the –l option.
Output
The output of this command is three files with 2 or 1 granules of the TATMS, and GATMO products.
Data Granules in Input and Output Files
|
Aggregations |
Total
granules |
Granules
per aggregation for each product |
Files
(containing full and partial aggregations) |
Hyperslabs in each raw data dataset for a full
aggregation |
|
Input aggregations |
10 |
1 |
10 |
1 |
|
Output aggregations |
10 |
2 |
3 (2 full, 1 partial) |
2 |
Notes
á This example uses a subset of the products in the example file. It would be possible to use entirely different products if they were also in the input files.
á The GATMO GEO product is associated with both SATMS and TATMS, and therefore will be packaged with both products or with either product alone.
Command Runtime Output
The status messages produced by the program during processing are listed below.
Produced 2 granules in
GATMO-TATMS_npp_d20120403_t2359123_e0000159_b02250_c20120831162351301821_XXXX_XXX.h5
Produced 2 granules in
GATMO-TATMS_npp_d20120404_t0000163_e0001199_b02250_c20120831162351458586_XXXX_XXX.h5
Produced 1 granules in GATMO-TATMS_npp_d20120404_t0001203_e0001519_b02250_c20120831162351559138_XXXX_XXX.h5