Summary of the proposed approach to ranking

[Nikolai Mansourov <nick@kdmanalytics.com>]

Hello all,

I have recently uploaded a document where I am proposing to extend SARIF with measurements for each result object. I propose to distinguish two kinds of measurements: tool-specific and standard (common) ones. Each measurement is a pair {value, measureId}. I propose to use the OMG Structured Metrics Metamodel (SMM) as the means to provide structured definitions of measures. This way both tool-specific  as well as standard measure definitions will be defined iusing the same set of objects. Both kinds of definitions are viewed as part of the metadata, similar to rules. As the result, each tool can publish their SMM-based definitions as part of their metadata, and the standard ones can be published by the community. 

This nicely splits the effort into two streams: 1) an exchange format (a stable, standards-based, evolution-proof format is proposed), and 2) the set of standard measures, which should be a long-term continuous discussion.

The current proposal focuses at the first concern. I would like to start the discussion regarding the second concern - the common set of measures - and to this end I have also posted a white-paper “Ranking Weakness Findings”, which frames the discussion and where I’m describing some objective measures as candidates into the common set.

There is already a community called Consortium for IT Software Quality (CISQ) that publishes definitions of common metrics using SMM.

The key design decision of the proposal is to align the SARIF measure definitions with SMM. I can think of two possible strategies: 1) full compliance to SMM - use SMM XMI to define measures; 2) partial compliance (alignment) where a JSON format is used to represent SMM elements.

I am recommending this partial compliance route, because 
1) this will allow the SARIF audience to better own the definitions (this approach removes the burden for a non-OMG audience to go through the SMM, MOF and XMI specifications to figure out what they need to emit).
2) the resulting definitions will fit with the rest of the JSON file format
3) I can focus on a smaller subset of SMM

So the proposal is a JSON rendition of SMM, with a full example and a section arguing compliance with SMM.

The full context for this proposal is provided in a white paper “Ranking weakness findings”. T

An even broader context is the discussion of an alignment between the OASIS SARIF and the OMG TOIF - available as a separate document, where the two specs are compared, and the roadmap is discussed.

Best regards,

Dr. Nikolai Mansourov,
CTO KDM Analytics,
OMG liaison to OASIS