/** @file spec.page */

/** @page page_spec_4 GROUND SUPPORT DATA FORMAT 

<hr>
<center>STOP!</center>

This controlling document for file formats and storage hieracy is at:

https://saturn.physics.uiowa.edu/svn/juno_hw/GSE/trunk/ground_proc/doc/waves_gs_fmts_stor.txt

The relevant sections from waves_gs_fmts_stor.txt are repeated here for ease
of access.  But should not be considered authoritative.
<hr>

<pre>
4 GROUND SUPPORT FILE FORMAT
============================
The format for data supplied from Waves to it's support systems is called the 
IDP (Instrument Dependent Protocol).  As development proceeds, IDP packets will
be supplied embedded in various headers.  For example the Lockheed Martin
Spacecraft Emulators (SCEs) provide low rate data as IP packets, wrapping UDP
packets, wrapping CIP packets, which contain the actual waves IDP data.

No matter the source of the data it is the task of the various processors to
output data streams in a standardized form.  This form is:

   1. A General Ground Support Packet Header.
   
   2. Instrument and Mission specific Headers.
   
   3. Instrument data.
   
   4. A copy of the total packet length.

</pre>
<hr>
@ref page_spec_4_1 "Next @>"

- @subpage page_spec_4_1
- @subpage page_spec_4_2
- @subpage page_spec_4_3
- @subpage page_spec_4_4
- @subpage page_spec_4_5
*/

/** @page page_spec_4_1 Ground Processing Intermediate Steps and Data Contents
<hr>
<pre>
4.1 Ground Processing Intermediate Steps and Data Contents
----------------------------------------------------------
The input data for ground processing is the Waves output stream along with any
headers assigned by "out of house" spacecraft or mission operations data
handling systems.  One of the main tasks of the ground processing software is
to collect data from an upstream source, for instance the Juno spacecraft
emulator, and to produce standardize uncalibrated data whose format is
independent of how the data was packaged for delivery by the Waves instrument.  

Waves packages data in a variety of ways in order to meet restrictions on packet
lengths and to efficiently utilize the available telemetry bandwidth.  Most of
the in-flight packaging must be undone on the ground to recover the collected
waveforms and spectra.   

Each ground packet consists of a header and a data section.  Initially the data
section is quite complicated, containing one IDP packet as transmitted from
Waves and thus is a non-trivial mixture of headers and in-flight packaged data.
Ground software then undoes most of the in-flight packaging.  At the point 
that uncalibrated data sets are generated by the ground software, the data 
section consists of only one waveform or one spectra in a simply readable
format.   

The following table lists the ground processing steps leading up to uncalibrated
data sets along with the data section contents after each step.  The routine
or program at each step attempts to output data packets in the Uncalibrated
state, i.e. one decompressed capture per packet.  Note that only science data
proceeds through all the steps.  Housekeeping data and other CIP types stop
after the reading stage.


  Step              Science Data Section Contents
  --------------    --------------------------------------
  Reading           1 Waves IDP containing 1-N mini-packets (HSK Stops Here)
  Unpacking         1 Mini-packet Data -or- 1 capture 
  Unsegmentation    1 Mini-packet Data -or- 1 capture 
  Decompression     1 Mini-packet Data -or- 1 capture 
  Deglopping        1 capture

The program or routine that outputs data in the Uncalibrated state assures that
all temporary fields in the Science Data Status Block (see 4.3.3.1) are zeroed
and sets the Data Contents Indicator (see 4.3.1) to 0x4 (1 Capture).

</pre>
<hr>
@ref page_spec_4 " @< Previous 4"

@ref page_spec_4_2 "Next 4.2 @>"
*/


/** @page page_spec_4_2 Ground Support Headers (GSE Header)
<pre>
4.2 Ground Support Headers (GSE Header)
---------------------------------------
Juno Waves ground support headers follow a general pattern that is reusable 
for other instruments.  

    +--------------------+
    |       Prefix       |
    +--------------------+
    |   Header Block 1   |
    +--------------------+
    |   Header Block 2   |
    +--------------------+
    ...
    +--------------------+
    |   Header Block N   |
    +--------------------+
    |  Optional Padding  |
    +--------------------+
    
The first 16 bytes of the header, called the prefix, provides a 32-bit sync 
pattern, a CRC, as well as the lengths of the header and data sections of the
packet.  It also contains the spacecraft ID which acts as as name space
indicator for various enumerations.   The prefix is followed by a series of
information blocks some of which are general and some of which are instrument
specific.

Padding consists of zero bytes, 0x00, and where possible 0x00 should never
indicate a filled in value.  For Waves, all headers must be padded out such 
that they are 256 bytes long.

</pre>
<hr>
- @subpage page_spec_4_2_1
- @subpage page_spec_4_2_2
- @subpage page_spec_4_2_3
- @subpage page_spec_4_2_4

@ref page_spec_4_1 " @< Previous 4.1"

@ref page_spec_4_2_1 "Next 4.2.1 @>"
*/


/** @page page_spec_4_2_1 Prefix
<hr>
<pre>
4.2.1 Prefix
------------
There is one prefix per packet, it is always 16 bytes long.  The first 4 bytes
are the sync pattern 0xFA6C2741.  This is useful in the case that a file
becomes fragmented or has invalid length information for one or more packets.
          
       +--------+--------+--------+--------+--------+--------+--------+--------+
000-007|  0xFA  |  0x6C  |  0x27  |  0x41  |       CRC       |  HEAD_TAIL_LEN  |
       +--------+--------+--------+--------+--------+--------+--------+--------+
008-015|        TOTAL_PACKET_LENGTH        |  SPACECRAFT_ID  | HDR_VER|  CONT  |
       +--------+--------+--------+--------+--------+--------+--------+--------+

CRC:
    16-bit unsigned MSB first integer containing the CRC-CCITT of everything
    that follows for this packet up to and including the trailing length bytes.
    See Appendix B for a definition of the CRC-CCITT algorithm in C.
  
HEAD_TAIL_LEN:
    A signed 16 bit MSB first integer containing the number of bytes in
    the GSE header - 1.  This length includes the sync pattern AND the trailing
    4 length bytes.  Thus the maximum number of non-data bytes in a packet is 2
    to the 16, or 65536 bytes.  
    
    NOTICE:  In general GSE Headers are not required to be fixed length and
    are not required to have any special minimum length above the 16 bytes 
    required for the prefix.  However, since zero fill bytes are ignored at
    the end of the header, and to aid in debugging broken files, all Waves 
    headers are REQUIRED to be have MINIMUM of 260 bytes for this field.
    This means there are a minimum of 256 bytes in the header plus 4 bytes
    for the trailing length. 

TOTAL_PACKET_LENGTH:
    An unsigned 32bit MSB first integer containing the number of bytes in
    the total packet -1.  Thus the maximum length of a packet including all
    data and non-data is 2 to the 32, or 4294967296 bytes.
    
SPACECRAFT_ID:
    A 16-bit MSB first signed integer. If available this is the NAIF spacecraft
    code.  For Juno this is -61 or 0xFF 0xC3 as hex bytes.  Note that the 
    SPACECRAFT_ID sets the name space or "code table" for block types, 
    program IDs, and the CONTent byte.
    
HDR_VER:   
    An unsigned byte.  This document describes waves ground support packet
    version 0x03.
 
CONT:
   An identifier for the contents of the packet.
   
   For Juno this byte consists of two parts, the upper 4 bits are the Data
   Kind Indicator, this is the same as the CIP ID, even if the data was
   gathered from a source with out CIP headers.  Thus from:  
   
     documents/trunk/Waves_IDC_11Juno08-1.doc
   
      0x6 - Down-link Data (Housekeeping)
      0x7 - Process Data (Info from waves to Juno CDS)
      0x8 - Science Data
   
   The lower 4 bits are the Data Contents Indicator, the values are:
      
      0x1 - Raw IDP
      0x2 - 1 Mini-packet
      0x3 - 1 Mini-packet's Data
      0x4 - 1 Capture
</pre>
<hr>
@ref page_spec_4_2 " @< Previous 4.2"

@ref page_spec_4_2_2 "Next 4.2.2 @>"
*/


/** @page page_spec_4_2_2 General Organization of Header Blocks
<hr>
<pre>
4.2.2 General Organization of Header Blocks 
-------------------------------------------
The rest of the header is series of Header Blocks, each of which may be up to
256 bytes in size and always start with a size byte and a block ID byte.  At a
minimum, this provides sufficient information for programs to skip over blocks
that are unrecognized.   

     Block    Process         
    Type ID   Type ID            Purpose
    -------   -------  ---------------------------------------------
      0x00       -     Never a legal block type
   
      0x10       -     Science data status
   
      0x70      0x01   Reader processing information
      0x70      0x02   Unpacker processing information
      0x70      0x03   Unsegmenter processing information
      0x70      0x04   Decompressor processing information
      0x70      0x05   Deglopper processing information
      0x70      0x06   Reformatter processing information
      0x70      0x07   Calibrator processing information
   
      0xA0       -     Program Error, Warning or Notice Number
      0xA7       -     Program Error, Warning or Notice Text Message
   
      0xD0       -     Aux. Information.
    
All programs shall output blocks into the header in ascending order by Block 
Type ID and then by Process Type ID if available.  This fixes block locations
within the header so that programs which only deal with a subset of all
possible content types may access various fields merely by absolute offset from
the start of the header.
</pre>
<hr>
@ref page_spec_4_2_1 " @< Previous 4.2.1"

@ref page_spec_4_2_3 "Next 4.2.3 @>"
*/


/** @page page_spec_4_2_3 Common Block Bytes
<hr>
<pre>
4.2.3 Common Block Bytes
------------------------
All blocks start with the following two bytes:

        +--------+--------+
r00-r01 |BLK_TYPE| BLK_LEN|
        +--------+--------+
  
BLK_TYPE - The Block Type ID.
  
BLK_LEN - An unsigned byte containing the number of bytes in the block
     minus one, thus the maximum block size is 256 bytes.
</pre>
<hr>
@ref page_spec_4_2_2 " @< Previous 4.2.2"

@ref page_spec_4_2_4 "Next 4.2.4 @>"
*/



/** @page page_spec_4_2_4 Common Processing Block Bytes
<hr>
<pre>
4.2.4 Common Processing Block Bytes
-----------------------------------
Processing blocks allow for storage of extra information associated with a 
particular data processing step.  All blocks with type id 0x70 start as follows:

       +--------+--------+--------+--------+--------+--------+--------+--------+
r00-r07|  0x70  | BLK_LEN| PROCESS| PROGRAM|           BEGIN_TIME_SEC          |
       +--------+--------+--------+--------+--------+--------+--------+--------+
r08-r15|BEG_TM_US_DIV_100| ERRORS |WARNINGS| NOTICES|
       +--------+--------+--------+--------+--------+

PROCESS
   An unsigned byte containing the identifier for the processing step.  Process
   IDs are spacecraft specific.
   
PROGRAM
   An unsigned byte containing the identifier for the program that handled the
   processing step.  See Appendix C for a list of program IDs.

BEGIN_TIME_SEC
   Unsigned MSB first 32-bit integer containing the time the process
   identified by the PROCESS byte was started.  It is the number of seconds
   elapsed since 00:00:00 on January 1, 1970, Coordinated Universal Time (UTC). 

BEG_TM_US_DIV_100 (Begin Microseconds Divided by 100):
   Unsigned MSB first 16-bit twos compliment integer containing 1/100th the
   number of microseconds since the begin time seconds.  To get the begin
   time microseconds, multiply this value by 100.  Thus the time resolution is
   0.1 milliseconds.

ERRORS
   The number of errors encountered during this processing step.  Associated
   error number or error message blocks should be included later in the header
   to provide error details.  

WARNINGS
   The number of warnings encountered during this processing step.  Associated
   warning number or warning message blocks should be included later in the
   header to provide warning details.
   
NOTICES
   The number of notices encountered during this processing step.  Associated
   notice number or notice message blocks should be included later in the header
   to provide details.  
</pre>
<hr>
@ref page_spec_4_2_3 " @< Previous 4.2.3"

@ref page_spec_4_3 "Next 4.3 @>"
*/
