What is the use case for timestamp_precision?
So for the sake of consensus I will follow... So just to be crystal clear, we are all saying:
The "timestamp_precision" field is a separate field at the same level as the "timestamp" field. It has the following valid options: year month day hour minute second
The default value for the "timestamp_precision" field is "second". So if the "timestamp_precision" fields is not present, the precision is to a "second". Any precision below the second is implicit in the number of fractional seconds in the "timestamp" field. So in code you will need to check for the presence of sub fractional values and then figure out how long the value is. And if it is found, then you will know that the precision is greater than a second. But there will be no value entered in to the "timestamp_precision" field. So in general, this field will most often be absent for any precision to a second or greater.
For the "timestamp" field, a timezone MUST be included and MUST be in "Z" Zulu time. So before you can send the data across the wire, you need to covert the time to UTC/Zulu time and send it that way. An example of 1:00 PM EDT would be: 2016-01-19T17:00:00.123456Z
The "T" and "Z" characters SHOULD be upper case.
Thanks,
Bret Bret Jordan CISSPDirector of Security Architecture and Standards | Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
This is a reason to say No. If you need place, make it a data element. Any inference of a data element from some other data element is a guarantee it will be wrong.
Worse yet, if we say, “This happened in New York” but the time stamp says “Z-06:00” - who do you believe?
For me I would vote for Precision 1 and Timezone 1.
Having the timezone offset also gives some implicit context of where the event took place in a follow the sun. This can give some additional context and understanding.
Thanks,
Bret Bret Jordan CISSPDirector of Security Architecture and Standards | Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
For the record, I didn’t care about nanoseconds – that was coming from other members of the community. Precision 1, Timezone 1 would be my vote I’d vote for Precision 1, Timezone 2. At the f2f, I don’t recall anyone other than Alex caring about precision less than a second (nanosecond). Joey -- Joey Peloquin, Senior Manager Citrix Security | Threat Intelligence and Vulnerability Management Citrix Systems, Inc. | 851 West Cypress Creek Road | Fort Lauderdale, FL 33309 My preference is Precision 1 Timezone 2, or +1 to John’s proposal. The only real issue is if the precision is less than a second, e.g., 1/10th minutes (6 seconds), 1/4 minutes (15 seconds), or even whole minutes. The question then is does anyone care? How about we put together concrete proposals for precision and for timezone? - (1): Separate precision field, values are: year, month, day, hour, minute, second (default = second). Any precision below the second is implicit in the number of fractional seconds in the timestamp field.
- (2): Separate precision field, values are: year, month, day, hour, minute, second, decisecond, centisecond, millisecond, microsecond, nanosecond (default = microsecond).
- (1): Timezone is required, may be any valid offset
- (2): Timezone is required, MUST be UTC (Z)
Do these cover the various proposals? I intentionally didn’t include proposals with not much support (ISO-8601 for precision, not requiring timezone). My read of the f2f was that we didn’t discuss anything relevant for the precision proposals but for timezone #2 is in line with the consensus. If we’re going to vote, my vote is Precision 1 and Timezone 2. In all honesty though at this point I think any of the proposals are workable. Time Zones specified with a Z and in UTC only. Arbitrary fractional second precision. One note at the end about time stamp precision. Time Zone Thing #1: Anything that cannot be unambiguously translated into UTC never, ever gets put on the wire. I.e., you can choose between T and an offset or Z. Once we agree on Thing #1, then we are looking at what our preference would be. We are expecting global usage, or at least usage spanning time zones. As such, no matter how one stores data, i.e, everything in UTC or everything in local time + UTC offset, the client will be doing time zone calculation to translate into local time for display or analysis. Given thing #2, from a global efficiency perspective, we might as well only allow UTC. The client sending the original message will need to know its local time zone anyway to properly format the offset. As such, it might as well do the calculation to send the time in UTC. That way the server needs no calculations of time zone offset. The retrieving clients are all likely to be doing time zone calculations no matter what time zone the server sends the time in, so there is no extra burden here. In fact, if the client is an analysis bot, it will want things in UTC anyway. If you want to employ Postel’s Law, feel free to accept offset (T) time. However, that will encourage bad behavior and should be discouraged. Senders MUST NOT send anything other than Zulu time. Parsers have to scan the entire time stamp to ensure we are not getting snared by fuzzing, buffer overflowing, insane values, etc. At that point, it does not matter whether we have fixed-length time stamps or variable-length time stamps. Various applications have various needs. For some, 15-second precision is more than adequate, like for physical security analysis applications. For others, sub-nanosecond precision is barely adequate, like for 10GE traffic flow analysis. Because today we are talking about applications needing between zero and nine digits of precision, and with 1TbE a future possibility, it makes neither sense to mandate a default precision nor precision indicators. Recall Precision Thing #1 above. Your parser will be scanning the entire time stamp string no matter what. You might as well read until you find the delimiting space. Both RFC 3339 and ISO 8601 specify arbitrary fractional second precision. Nothing new here. If you really believe scanning the time stamp to read the fractional number of seconds is computationally a time suck, sponsor a tiny bit of work and we’ll do an open source implementation that will blow doors off you bad implementations ;-) No specified limits or markings for fractional seconds. Does anyone care about the measured precision being transmitted along with the data? For example, is 12:34:56.000000 really saying that, to the microsecond, it was 12:34:56, or is it just saying that to the nearest second that was the time? Conversely, does 12:35:00 mean precisely at 12:35, about 12:35, or somewhere between 12:34:30 and 12:35:29? Again, do we care? Eric Berger has very strong opinions about this, and should be back online next Monday. Lets make sure we give him time to comment. But, for everyone else, please speak up. We really need to put this to bed soon. Bret Jordan CISSP Director of Security Architecture and Standards | Office of the CTO PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." There is nothing to prevent us from restricting the use of RFC 3339 datetime’s to UTC. We CAN do it and I believe the consensus at the F2F was that we should. So requiring the timestamps to be in UTC is divergent from RFC 3339. RFC 3339 requires that times be derived off of UTC, which is what I think we all want. Meaning you can NOT do this: 2016-01-18T11:10:10 and expect people to think that it is EDT. That would and should FAIL. You would need to do: 2016-01-18T11:10:10-04:00 for EDT or you could do 2016-01-18T15:10:10Z / 2016-01-18T15:10:10+00:00 I think if we are going to go down the path of RFC 3339, regardless of my previous comments, we should just use it as is. This will hopefully guarantee that libraries will work. The "timestamp" field in STIX, CybOX, and TAXII MUST use an RFC 3339 compliant timestamp that includes a timezone offset from UTC or the value is in UTC with a "Z". Examples: 2016-01-18T11:10:10-04:00 2016-01-18T11:10:10.123456-04:00 This means you can include as many or as few fractional sections as you want. The "timestamp_precision" field will be at the same level as the "timestamp" field and will be optional. The default precision is Microseconds. This field in JSON is a string field and has the following valid options in the ENUM: Director of Security Architecture and Standards | Office of the CTO PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." The more I thought about this the more my thinking was driven by what various datetime libraries allow the developer to do. While in a perfect world I’d prefer the ISO 8601 “just specify what you know” (which allows 2016-01-01 for example to say January 1, 2016), libraries that support 8601 don’t seem to give the developer any way of distinguishing between “2016-01-01” and “2016-01-01T00:00:00.0Z”. Therefore, we’re better off going with RFC 3339 which mandates a full date/time and the separate precision specifier we’ve previously discussed. As far as fractional seconds, I say we allow zero or more digits – the libraries support that, it’s compliant with RFC 3339 and at the F2F no one could articulate a reason why that was a burden to implementers. All times expressed in UTC (“Z”) Separate precision specifier Thoughts??? We really need to decide this. This along with IDs are the next major things we need to resolve and resolve SOON. Director of Security Architecture and Standards | Office of the CTO PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." As this issue was brought up at the Face2Face, and thus reopened for debate and consensus, and the consensus today was leaning toward the use of RFC3339 as is, without further stipulation and the removal of timestamp_precision field, I thought it would be best to re-read the RFC for clarification. Section 4.2. Local Offsets Allows for local time, so long as it is an offset from UTC 1996-12-19T16:39:57-08:00 I do not see anything in RFC3339 that allows for dates/times with less precision than a fully qualified timestamp: yyyymmddThh:mm:ssZ ISO8601 does explicitly allow for reduced precision by just truncating the value. From my understanding of the community we have the following requirements: 1) Represent reduced precision 2) Represent an arbitrary number of fractional sections 3) Represent values in local time as a offset to UTC or in UTC 20150114T23:21:20.123456Z 20150114T23:21:20.123456-08:00 Director of Security Architecture and Standards | Office of the CTO PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
|