RE: [sarif] Important subtlety in region definition (long)

["Larry Golding \(Comcast\)" <larrygolding@comcast.net>]

I pushed a revision to the change draft that adds this section:

3.22.4 Independence of text and binary regions

The text-related and binary-related properties in a region object SHALL be treated independently. That is, the value of a text-related property SHALL NOT be inferred from the value of any set of binary-related properties, and vice versa.

EXAMPLE: This example is based on the sample text file show in NOTE 1 of §3.22.2. It represents invalid SARIF because the text-related and binary-related properties are inconsistent. At first glance they appear to be consistent because the byte at offset 2 is indeed on line 1:

{ "startLine": 1, "byteOffset": 2, "byteLength": 6 }

However, because the default values for the missing text-related properties are determined entirely from the existing text-related properties, and independently of any binary-related properties, this region is in fact equivalent to this one:

{

  "startLine": 1,

  "startColumn": 1,  // Missing startColumn defaults to 1.

  "endLine": 1,      // Missing endLine defaults to startLine.

  "endColumn": 6,    // Missing endColumn defaults to (length of endLine + 1),

                     // exclusive of newline sequence.

  "byteOffset": 2

  "byteLength": 6

}

This makes it clear that the text-related and binary-related properties represent different ranges of bytes, and therefore the region is invalid.

Larry

 

From: sarif@lists.oasis-open.org <sarif@lists.oasis-open.org> On Behalf Of Larry Golding (Comcast)
Sent: Friday, May 25, 2018 1:51 PM
To: sarif@lists.oasis-open.org
Cc: 'James A. Kupsch' <kupsch@cs.wisc.edu>; 'Michael Fanning' <Michael.Fanning@microsoft.com>; 'Luke Cartey' <luke@semmle.com>
Subject: RE: [sarif] Important subtlety in region definition (long)

 

Hearing no objection, and having gotten agreement offline from Michael, I’m going to make this restriction explicit in the change draft for #93.

 

Larry

 

From: sarif@lists.oasis-open.org <sarif@lists.oasis-open.org> On Behalf Of Larry Golding (Comcast)
Sent: Thursday, May 24, 2018 9:31 AM
To: sarif@lists.oasis-open.org
Cc: 'James A. Kupsch' <kupsch@cs.wisc.edu>; Michael Fanning <Michael.Fanning@microsoft.com>; Luke Cartey <luke@semmle.com>
Subject: [sarif] Important subtlety in region definition (long)
Importance: High

 

Please read to the end, where I pose two questions.

 

The new change draft for Issue #93, “Problems with regions”, addresses concerns raised by Jim and Luke, and incorporates some nice design suggestions from Jim.

 

The spec now says that a single region object can represent both a “text region” (a contiguous sequence of characters) and a “binary region” (a contiguous sequence of bytes), using separate sets of properties. And the spec says that if a region object represents both a text region and a binary region, then the text-related properties and the binary-related properties must represent exactly the same range of bytes.

 

The spec does not allow you to specify a region by a mixture of text- and binary-related properties. For example, consider a UTF-16 file with no BOM and contents "abcde\r\n", and consider a region that includes the characters "bcd". The spec allows you to represent this region in many ways, such as:

 

Text-related line/column properties:

 

{ "startLine": 1, "startColumn": 2, "endColumn": 5 }

 

Text- related offset/length properties:

 

{ "charOffset": 1, "charLength": 3 }

 

A mixture of text- related line/column and offset/length properties:

 

{ ""startLine": 1, "startColumn": 2, "charLength": 3 }

 

Binary-related offset/length properties:

 

{ "byteOffset": 2, "byteLength": 6 }

 

But the spec does not allow this:

 

{ "startLine": 1, "byteOffset": 2, "byteLength": 6 }  # INVALID

 

I could have written the spec to allow this, but I chose not to, for simplicity. The spec already has paragraphs of text and dozens of examples illustrating valid combinations of the text-related properties alone. I judged that it would be too difficult to express, and too difficult for implementers to understand and implement correctly, language that attempted to enumerate all legal combinations of text-related and binary-related properties. Instead, I required each set of properties to stand alone, and for them to be consistent.

 

As the spec stands, that mixed region above would be equivalent to:

 

{

  "startLine": 1,

  "startColumn": 1,             // Missing startColumn defaults to 1.

  "endLine": 1,                 // Missing endLine defaults to startLine.

  "endColumn": 6,               // Missing endColumn defaults to (length of endLine) + 1, exclusive of newline sequence.

 

  "byteOffset": 2

  "byteLength": 6

}

 

… and now the text-related properties and the byte-related properties represent different byte ranges.

 

My two questions are:

 

1. Do you agree with my proposal to treat the text-related properties and binary-related properties separately?
2. If so, should I state that explicitly, and give this example?

 

*sigh* Having written all this, I guess the answer to #2 has to be “Yes” if the answer to #1 is “Yes”.

 

Thanks,

Larry