Table 7-06 Level of Data

[amilan17]

Initial request

During the FT2023-1 NFP review, we received feedback that "CIMO" is still referenced in the descriptions of codes in table 7-06 for Level of Data.

Amendment details

updated 30 June 2023

notation	name	description	New description
level0	Level 0	CIMO: Analogue/digital electric signals.	Instrument and auxiliary data reconstructed from satellite raw data after removing communications artefacts. [source: GIMO (WMO-No. 8), volume IV]
		CEOS: Physical information: Reconstructed unprocessed instrument data at full space time resolution with all available supplemental information to be used in subsequent processing (e.g. ephemeris, health and safety) appended.
level1	Level I	CIMO: Level I data (Primary Data): in general, are instrument readings expressed in appropriate physical units, and referred to Earth geographical coordinates. They require conversion to the normal meteorological variables (identified in Part I, Chapter 1). Level I data themselves are in many cases obtained from the processing of electrical signals such as voltages, referred to as raw data. Examples of these data are satellite radiances and water-vapour pressure, positions of constant-level balloons, etc. but not raw telemetry signals. Level I data still require conversion to the meteorological parameters specified in the data requirements.	Level I data, in general, are instrument readings expressed in appropriate physical units, and referred to with geographical coordinates. They require conversion to the normal meteorological variables (identified in Part I, Chapter 1). Level I data themselves are in many cases obtained from the processing of electrical signals such as voltages, referred to as raw data. Examples of these data are satellite radiances and water-vapour pressure. [source: GIMO (WMO-No. 8) , volume V]
		CEOS: Physical information: Unpacked, reformatted level 0 data, with all supplemental information to be used in subsequent processing appended. Optional radiometric and geometric correction applied to produce parameters in physical units. Data generally presented as full time/space resolution. A wide variety of sub level products are possible.	Instrument data extracted, at full original resolution, with geolocation and calibration information. [source: GIMO (WMO-No. 8), volume IV]
level2	Level II	CIMO: Level II data (Meteorological parameters). They may be obtained directly from many kinds of simple instruments, or derived from Level I data. For example, a sensor cannot measure visibility, which is a Level II quantity; instead, sensors measure the extinction coefficient, which is a Level I quantity. NOTE: Data exchanged internationally are Level II or Level III data.	The data recognized as meteorological variables are Level II data. They may be obtained directly from instruments (as is the case for many kinds of simple instruments) or derived from Level I data. For example, a sensor cannot measure visibility, which is a Level II quantity; instead, sensors measure the extinction coefficient, which is a Level I quantity. [source: GIMO (WMO-No. 8) , volume V]
		CEOS: Geophysical information. Retrieved environmental variables (e.g., ocean wave height, soil moisture, ice concentration) at the same resolution and location as the level 1 source data.	Geophysical product retrieved from a single instrument in the original projection. [source: GIMO (WMO-No. 8), volume IV]
level3	Level III	CIMO: Level III (Initial state parameters) are internally consistent datasets, generally in grid-point form obtained from Level II data by applying established initialization procedures. NOTE: Data exchanged internationally are Level II or Level III data.*	Level III data are those contained in internally consistent datasets, generally in grid-point form*. They are not within the scope of the present Guide. Data exchanged internationally are Level II or Level III data. [source: GIMO (WMO-No. 8) , volume V]
		CEOS: Geophysical information. Data or retrieved environmental variables which have been spatially and/or temporally re-sampled (i.e., derived from level 1 or 2 products). Such re-sampling may include averaging and compositing.	Geophysical product retrieved from a single instrument, mapped on uniform space and time grid scales, possibly on a multi-orbital (for LEO) or multi-temporal (for GEO) basis. Irreversible process due to resampling. [source: GIMO (WMO-No. 8), volume IV]
level4	Level IV	CEOS: Thematic information. Model output or results from analyses of lower level data (i.e., variables that are not directly measured by the instruments, but are derived from these measurements).	Composite multi-sensor and/or multi-satellite products; or result of model analysis. [source: GIMO (WMO-No. 8), volume IV]
raw	Raw	CEOS: Physical information: Data in their original packets, as received from a satellite
unknown	unknown	The value is unknown or missing.
inapplicable	inapplicable	None of the codes in the table are applicable in the context of this particular observation.

Comments

No response

Requestor(s)

James Lunny, NFP for Codes and Data Representation Matters
Met Service, New Zealand

Stakeholder(s)

Enter list of stakeholder(s).

Publication(s)

Example: Manual on Codes (WMO-No. 306), Volume I.3, WMO Codes Registry, Code table 7-06, Level of Data,http://codes.wmo.int/wmdr/LevelOfData

Expected impact of change

LOW

Collaborators

No response

References

No response

Validation

No response

[joergklausen]

I am not sure we ought to change anything here. The fact that CIMO does no longer exist doesn't change the fact that it was CIMO who generated these elements. After all, we also reference Newton in our unit table ...

[amilan17]

https://github.com/wmo-im/tt-wigosmd/wiki/2023.04.21-TT-WIGOSMD notes:
If the GIMO Guide has the definitions, then we could update these with proper references. Gao noted that there are other NASA definitions at https://www.earthdata.nasa.gov/engage/open-data-services-and-software/data-information-policy/data-levels and Anna noted that NOAA has different/similar definitions.

[amilan17]

https://github.com/wmo-im/tt-wigosmd/wiki/2023.05.04-TT-WIGOSMD notes:
@meulenvd is still checking the definitions in the regulatory material of WMO.

[fierz]

@joergklausen @meulenvd @gaochen-larc
I support better defining the source of such definitions, for example WMO-No. 8 (Guide to instruments and methods of observation). Also, I feel like the above table includes both levels for in situ instruments (ground based) as well as for satellite data. These two use cases may have to be treated separately?

[amilan17]

https://github.com/wmo-im/tt-wigosmd/wiki/2023.06.01-TT-WIGOSMD notes:

• Jitze noted that the Manual on WIGOS and GIMO also refers to levels and suggests that an option is to simplify the descriptions and/or bring this to the various groups to get consensus;
• Joerg noted that the original request is noting that CIMO is referenced which no longer exists and these definitions were valid at that time;
• @meulenvd review GIMO definitions and if they are different, then add new GIMO definitions;

[meulenvd]

The real issue is that there are many sources. The reference to CEOS and CIMO is arbitrary, out-of-date or obsolete. Also the definitions are not valid anymore.
A source, containing definitions with a mandatory status is the list of definitions in WMO-No. 49 (Technical Regulations), Vol. I (see below).
Other sources within the WMO set of Manuals and Guides: WMO-No. 8 (GIMO), Vol. IV, par. 2.3.2.6 and Vol. V, par. 3.1.1, and also WMO-No. 488, Part IV, par. 4.3. (Note: also the old (2013) Man. on the GDPFS WMO-No. 485, Vol. I contains a set of definitions, but removed later on).
Apart from there references, the WMO Space Programme (WSP) published definitions (comparable to GIMO, Vol. IV, par. 2.3.2.6) and the SPICE Programme* published a report (IOM 131, page 128), with definitions which can be considered as a future update for the definitions in GIMO Vol. V par. 3.1.1. These definitions don't differ very much from those expressed by the satellite community and I expect that further agreement to obtain one single set for both space as surface based observations within a short while. Also, the definitions in GIMO Vol. IV, par. 2.3.2.6 are very comparable to definitions expressed by WSP, NASA EOSDIS/Earthdata, This can be arranged within the INFCOM editing board working on updates of the GIMO.

The remaining issue is: do we like extensive definitions or short style, like in WMO-No. 47 or WSP? Both is possible and depends on what the users (readers) prefer.

Conclusion:

• easy way: CIMO and CEOS to be replaced by GIMO Vol. V, par. 3.1.1 and GIMO Part IV, par. 4.3, respectively. A bit strange, not correct, because the definitions themselves are obsolete.
• Better: to update the definitions according to what is published in GIMO (see reference below).
• Even better: To merge both definitions into one single set (within GIMO), taking into account the definitions stated by SPICE. Personally, I think that this is good possible and to be arranged within a short period.

Reference, WMO-No. 8 (GIMO), Vol. IV and V:

• WMOdata_levels.docx

Hyperlinks:

• WMO Space Programme: https://community.wmo.int/activity-areas/wmo-space-programme-wsp/data-products
• NASA EOSDIS: https://www.earthdata.nasa.gov/engage/open-data-services-and-software/data-information-policy/data-levels
• CEOS, Glossary of Acronyms and Terms, ver. 1.3, July 2020 (see page 22): https://ceos.org/document_management/Working_Groups/WGISS/Interest_Groups/Data_Stewardship/White_Papers/EO-DataStewardshipGlossary.pdf
• SPICE Final Report (see page 128): https://library.wmo.int/doc_num.php?explnum_id=5686 (see also: P3(10)_Reverdin_TECO2016.pdf)

Extracts from WMO-No. 49 (Tech. Reg.):

DEFINITIONS

• Level I data. In general, these data are instrument readings expressed in appropriate physical units; they are referred to using geographical coordinates. They require conversion to the normal meteorological and other variables of the physical environment. Level I data are in many cases obtained from the processing of electrical signals such as voltages, referred to as raw data.
• Level II data. The data recognized as meteorological or other variables of the physical environment. They may be obtained directly from instruments (as is the case for many simple instruments) or derived from Level I data.

---

*) SPICE stands for WMO Solid Precipitation Intercomparison Experiment (ask Rodica for more details on SPICE)

[amilan17]

If there are no updates since last meeting, then I suggest moving this to FT2024-1.

[amilan17]

https://github.com/wmo-im/tt-wigosmd/wiki/2023.06.30-TT-WIGOSMD notes:

• the team discussed options proposed by Jitze (see comment above); Joerg recommends creating a new codes list referencing modern definitions because the new definitions change the meaning of the levels and previous usage of the codes may still be relevant; 
• DECISION: follow through with Jitze's second recommendation "to update the definitions according to what is published in GIMO"; 
• @amilan17 will update issue summary and branch with information provided in the reference document that Jitze uploaded;

[amilan17]

@wmo-im/tt-wigosmd, @meulenvd  - I updated the issue summary with the details of new definitions from the GIMO publications, but I cannot find a definition of raw. Text in bold indicates the same content in old and new definitions. I'd like to get your comments before I update the branch.

[joergklausen]

@wmo-im/tt-wigosmd, @meulenvd - I updated the issue summary with the details of new definitions from the GIMO publications, but I cannot find a definition of raw. Text in bold indicates the same content in old and new definitions. I'd like to get your comments before I update the branch.

Found a reference: https://www.iso.org/obp/ui/#iso:std:iso:19165:-2:ed-1:v1:en, 3.8 Product Level

[amilan17]

https://github.com/wmo-im/tt-wigosmd/wiki/2024.01.18-TT-WIGOSMD notes:
@meulenvd -- Do you have feedback on proposal as it stands today? (see issue summary at the top). If these changes are approved by you and the team within a week or two, we can include this proposal in FT2024-1.

[meulenvd]

https://github.com/wmo-im/tt-wigosmd/wiki/2024.01.18-TT-WIGOSMD notes: @meulenvd -- Do you have feedback on proposal as it stands today? (see issue summary at the top). If these changes are approved by you and the team within a week or two, we can include this proposal in FT2024-1.

After modification of the Technical Regulations (and the Annexes, like the Manual on the WIGOS and the Manual on the GDPFS), the definition of Level of Data in now in the GIMO (WMO-No. 8), Guide to Instruments and Methods of Observations. However two definitions are given, one in the Chapter on Satellite observations (Vol IV) and the other in Vol V (data reduction, but largely on Surface based observations). This discrepancy between both definitions will be solved.
During the last SC-MINT Edititorial Board meeting (28 to 30 November 2023) is was decided to formulate a proposal to prepare a recommendation for the next INFCOM in order to get one uniform definition,

For the (short) time being we can accept the two definitions for Level of Data, One for surface based observations (ref. GIMO Vol. V), the other for Satellite based observations (ref. GIMO Vol IV), as stated on top under new description.

[amilan17]

https://github.com/wmo-im/tt-wigosmd/wiki/2024.02.15-TT-WIGOSMD notes:
(Jitze) current proposal will help with alignment with the WMO publications; @meulenvd update branch;

[JREyre]

@meulenvd @joergklausen @amilan17
I've checked all this. The inputs originating from CEOS and from CIMO sometimes address different aspects of the data but, as far as I can see, they are not incompatible. I support Jitze's suggestion that we await the outcome of the SC-MINT initiative to formulate a unified definition (with the aim that it is acceptable to both space-based and surface-based observation communities, and to communities involved in the development/use of L3 and L4 products).

[meulenvd]

@joergklausen @amilan17 @JREyre
The issue is that all references are out-of-date (and descriptions cannot be been found anymore), so the up-to-date references should be chosen as a first step. The follow up will be to present a proposal to SC-MINT with a uniform definition for both surface as space based observational data (to be published in WMO-No. 8). However, I we feel that the priority of this issue is really low, then we can wait of course until SC-MINT approved the proposal, but that may take longer than expected.

For this issue is see that for "raw", "unknown" and "inapplicable" there is no alternative which is trivial for "unknown" and "inapplicable", but there is some conflict with "raw" w/r "Level 0". For satellite based data "raw" is associated with the data packets or streams as received (complete or in fractions with erroneous data segments and unprocessed, just the pure data stream as recorded). For direct measurements, "raw" is associated with the analogue electric 'signals' (resistance, current, frequency) as the direct input to the measuring device, so not directly a data stream. As soon as it is digitized, but unprocessed and expressed in a unit it is qualified as "Level 0". Anyway, CEOS, NASA and WMO documents don't speak in terms of raw data, only in Level 0. Moreover I cannot imagine that "raw" data is disseminated as such, but only "Level 0". But if we want to keep "raw"in the table I propose to modify the definitions of both "Level 0" en "raw", to become:

• Level 0: Instrument and auxiliary data reconstructed from satellite raw data after removing communications artefacts. [source: GIMO (WMO-No. 8), volume IV] or the rawest and unprocessed output from an instrument or instrument transducer in native units (e.g. voltage) [source: GIMO (WMO-No. 8), volume V]
• raw: Unprocessed data in their original packets, as received from a satellite, or the unprocessed signals produced by measuring equipment. Note: usually the rawest output from an instrument or instrument transducer expressed as data in native units is qualified as Level 0 data.

[amilan17]

Thank you for your input.

[markusfiebig]

A comment from the side. When creating the ACTRIS vocabulary, we had a very similar discussion. We faced the situation that definitions of data levels vary from community to community, even among our own topic specific data centre units. This is caused by data treatment steps being distributed differently on the data levels in different communities. We decided not to use data levels at all towards the users (to avoid definition confusion), but to define vocabulary describing explicitly what type of treatment the data have received. Examples:

• https://vocabulary.actris.nilu.no/actris_vocab/timeliness
• https://vocabulary.actris.nilu.no/actris_vocab/dataqualitycontrol
• https://vocabulary.actris.nilu.no/actris_vocab/qualitycontrolmeasuretype
• https://vocabulary.actris.nilu.no/actris_vocab/qualitycontrolmechanism

[JREyre]

@markusfiebig @joergklausen @amilan17 @meulenvd
Thanks, Marcus. My comment is not related to Level of Data, but I was interested and pleased to see that ACTRIS uses the term "timeliness" (rather than "latency"). For several years, I've been conducting a campaign to remove "latency" from WMO documents and replace it with "timeliness". Some progress, but I fear that it will be a lost battle in the end!

[amilan17]

This issue is still under discussion.

[joergklausen]

@markusfiebig I commend ACTRIS and NILU for the vocabulary server, well made and something WMO should also look into. However, as TT-WIGOSMD we should still define the concept of Data Level and provide a few of the well accepted code lists. The concept exists, it obviously makes sense for some important communities if not for all of them.
@meulenvd I recommend you conclude the work on this issue, so that we can draw a line at the next meeting, please.

[joergklausen]

@meulenvd Can you please prepare this for our next meeting? On the new Level 0 definition, I want to raise a concern, namely that it only addresses satellite data : "Instrument and auxiliary data reconstructed from satellite raw data after removing communications artefacts. [source: GIMO (WMO-No. 8), volume IV]" I hope this is not the final word ...

[amilan17]

https://github.com/wmo-im/tt-wigosmd/wiki/2024.04.09%E2%80%90TT%E2%80%90WIGOSMD notes:
@amilan17 sent an email to Isabelle and Ercan requesting of the status of the GIMO updates; Joerg would like this issue closed in FT2024-2;

[meulenvd]

Status update:
A new proposal is drafted and submitted to the SC-MINT EdBd for follow up. I have chosen for a two step approach:

1. to define Levels of Data for surface based and for satellite based data apart
2. to merge these definitions later on to be valid for both type of data.

The proposal (new table see DataLevelsTable-Proposal.docx) is for step 1, where also definitions for surface based data are used from WMO IOM report 131 (Report of the WMO Solid Precipitation Intercomparison Experiment, “SPICE”, 2012-2015), with definitions which are well documented and also based on Data Levels for satellite data but suitable for surface based data.
The plan is now to consult Expert Teams (e.g. JET-OWR) which may have their own definitions for data levels.

[joergklausen]

@meulenvd latest proposal for level 0
Surface based data:
The rawest and unprocessed output from an instrument or instrument transducer in native units (e.g. voltage) (sources WMO-No. 8, volume V & WMO IOM report 131, ‘SPICE’)
Satellite based data: Instrument and auxiliary data reconstructed from satellite raw data after removing communications artefacts. [source: GIMO (WMO-No. 8), volume IV]

[amilan17]

https://github.com/wmo-im/tt-wigosmd/wiki/2024.05.16%E2%80%90TT%E2%80%90WIGOSMD notes:
The team looked at the proposed definitions in the attachment in Jitze's comment above.
it was noted that surface based observations that entail a retrieval can have a level one or two.

[joergklausen]

TT-WIGOSMD noted that proposal by @meulenvd for the surface based observations is very much centered (and thus limited) to meteorological observation, while most of the descriptions would apply to other disciplines as well. It was also recognized that for most surface-based in-situ measurements, level1 is not relevant, with the exception of a few cases where a retrieval is involved.

[amilan17]

https://github.com/wmo-im/tt-wigosmd/wiki/2024.06.20%E2%80%90TT%E2%80%90WIGOSMD notes:
DECISION - remove from FT2024-2 milestone and wait for updates from the editorial boards or curators of the publications with these definitions;

[david-i-berry]

Note, definitions are also included in the WMO-No. 49 (see https://library.wmo.int/idviewer/35722/16).