Re: [cti] Timestamp Serialization Question

["Jason Keirstead" <Jason.Keirstead@ca.ibm.com>]

It is simple and there isn't room for ambiguity - the window is defined as the precision field, on both sides of the provided timestamp. If I provide you 12:00:00 with 1 minute precision, then it means any time from 11:59:00 to 12:01:00.

This is how all time critical systems operate. For example, when you grab packets off of a 10G interface, the timestamps give you a value in nanoseconds, but if you read the documentation, the value is normally +/- some level of nanoseconds (10 to 100 or more, depending on hardware and other situations) - it is a window on either side of the value given. It's not the value given is the floor or the ceiling - the value is the center of the confidence interval and you have to go on either side. For STIX to buck this trend and say our timestamps are specifying the floor of the confidence interval would be very strange and counter-intuitive.


-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


Eric Burger ---01/19/2016 10:48:43 PM---This is another violent agreement, “yes, and” situations. Yes, this is how the data gets generated i

From: Eric Burger <Eric.Burger@georgetown.edu>
To: cti@lists.oasis-open.org
Date: 01/19/2016 10:48 PM
Subject: Re: [cti] Timestamp Serialization Question
Sent by: <cti@lists.oasis-open.org>

---

This is another violent agreement, “yes, and” situations.

Yes, this is how the data gets generated in the wild.

The problem is that unless we put our foot down and chose whether the time is the midpoint of the bucket or the bottom of the bucket, the consumer HAS NO CLUE what the bucket is. It is really trivial if you wrote the producer and the consumer: they both will encode your world view. It is really hard for a multivendor solution to have the same interpretation of what the bucket is unless we specify it here.

I really do NOT want to add yet another timestamp parameter. Precision is bad enough. “Error bars” or “count from the bottom” or “count from the middle” is really ugly. I would put the onus on the client and specify one and only one way to express time stamps.

On Jan 19, 2016, at 9:41 PM, Jason Keirstead <Jason.Keirstead@ca.ibm.com> wrote:


So, we could do it that way - which would require the producer to take the equivalent of 100% of their known precision and adjust their timestamps downward accordingly. I would argue strongly though that this is pretty much *never* how this is done in industry and would result in confusion. Normally the onus is on the consumer of information to interpret the producers information as they see fit when they know the precision.

Here is the difference:

- If I follow the specification below, and I read the time 12:00:00 off the clock and know my precision to be minute-level, then I would have to supply a timestamp of 11:59:00 with a precision of 1 minute ( note here the importance, that minute-level precision is not the same as 60 second precision - it actually requires a 2x the confidence interval time-boxing - this is important!). The consumer would then take that information and know "OK the time starts at 11:59:00 and ends between then at 12:01:00"

- The way it is normally done instead, is the producer of the time-sensitive information just sends whatever time came off of their information producing source. The consumer of that information then constructs the time-box around whatever rules they see fit. To carry forward the above example, the producer would send me 12:00:00 with 1 minute precision, and I would know implicitly, if I care about this at all, that that event could have occurred any time between 11:59:00 and 12:01:00.

I think that the second method is how pretty much all systems behave. I have never known a system to behave the first way.

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


<graycol.gif>Eric Burger ---01/19/2016 10:06:48 PM---I would offer the important precision is not not hours, minutes, or seconds, but number of seconds.

From: Eric Burger <Eric.Burger@georgetown.edu>
To: cti@lists.oasis-open.org
Date: 01/19/2016 10:06 PM
Subject: Re: [cti] Timestamp Serialization Question
Sent by: <cti@lists.oasis-open.org>

---




I would offer the important precision is not not hours, minutes, or seconds, but number of seconds. We also need to define whether the timestamp represents the middle of the range or the bottom.

For example, if I only transmit the hour portion of the timestamp of an event, then 12:00:00Z means anything from 12:00:00.000000000 to 12:59:59.999999999. However, if I transmit the closest hour portion of the timestamp of an event, then 12:00:00Z means anything from 11:30:00.000000000 to 12:29:59.9999999999.

Note that a typical data collection window is tenths of minutes. That is six seconds, not ‘seconds.’ I.e. 12:00:00Z means either 12:00:00Z - 12:00:05.999999999 or 11:59:57Z - 12:00:02.999999999.

My suggestion is (1) the timestamp represents the bottom of the range of the bucket and (2) the number is that for precision of less than a second (i.e., granularity of more than a second) is a bucket of seconds starting from the timestamp value with a precision number of seconds. So, examples would be:

12:00:00Z = event happened between 12:00:00Z - 12:00:00.9999999999Z [default precision = 1s]
12:00:00Z (precision = 60s) = event happened between 12:00:00Z - 12:00:59Z [formerly known as “minute precision”]
12:00:00Z (precision = 3600s) = event happened between 12:00:00Z - 12:59:59Z [formerly known as “hour precision”]
12:00:00Z (precision = 6s) = event happened between 12:00:00Z - 12:00:05.999999999Z [how else would you specify “tenth of a minute”?]

On Jan 19, 2016, at 8:51 PM, Jason Keirstead <Jason.Keirstead@ca.ibm.com> wrote:

The use case as I understand it at a high level is so that when someone submits a timestamp of 12:00:00 zulu, we know the difference between if they truely mean exactly 12:00:00 on the button, or if they only have second level precision available to them. And this is required because we aren't mandating a fixed format, but RFC 3339 which is variable.




Eric Burger --- Re: [cti] Timestamp Serialization Question ---


From:	"Eric Burger" <Eric.Burger@georgetown.edu>
To:	cti@lists.oasis-open.org
Date:	Tue, Jan 19, 2016 9:30 PM
Subject:	Re: [cti] Timestamp Serialization Question

---


I’m still clueless as to the use case.

Not a negative statement, but I would like to see the concise reason we need ‘precision’ before I weigh in, if at all.
On Jan 19, 2016, at 3:22 PM, Jordan, Bret <bret.jordan@BLUECOAT.COM> wrote:

There tends to be two options for dealing with objects that have multiple timestamps and their corresponding precision. Sean and I have been talking through the pros and cons of these. We would like to get everyone's opinion. Which do you prefer, option 1 or option 2



Option 1:
This option put the burden on the JSON serialization format to add an extra "_precision" field to each timestamp enabled field. This is a much flatter and easier to parse and process representation, but the con is it requires unique field names.
{
"type": "incident",
"initial_compromise_time" : "2015-12-07T22:00:00Z",
"initial_compromise_time_precision": "hour",
"first_data_exfiltrated_time" : "2015-12-09T05:11:00Z",
"first_data_exfiltrated_time_precision" : "minute",
"incident_opened_time" : "2016-01-15T11:19:22Z",
"incident_closed_time" : "2016-01-19T17:24:017Z"
}



Option 2:
This option will require a nested object and struct to store this data and will have an extra layer of indirection for all of those times when the timestamp is at the default precision.
{
"type": "incident",
"initial_compromise_time" : {
"timestamp": "2015-12-07T22:00:00Z",
"timestamp_precision": "hour"
},
"first_data_exfiltrated_time" : {
"timestamp": "2015-12-09T05:11:00Z",
"timestamp_precision" : "minute"
},
"incident_opened_time" : {
"timestamp": "2016-01-15T11:19:22Z"
},
"incident_closed_time" : {
"timestamp": "2016-01-19T17:24:017Z"
}
}



Thanks,

Bret



Bret Jordan CISSP
Director of Security Architecture and Standards | Office of the CTO
Blue Coat Systems
PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."