Question....... How far does the precision requirement really need to go? I know everyone always wants infinite flexibility.... But..... Perhaps at the end of the day, the only things that really matters is to know if the precision is to a given day or greater than a day.
Let me explain....
Put on my analyst hat..... I am guessing that something happened on my network last week some time. I think it happened on 2015-11-18.. With that amount of vagueness, that should be sufficient. And tools and other analyst can only work with that...
If I know that it for sure happened on 2015-11-18 and I am only guessing at the hour or minute, does it really matter if some of the time elements are zeroed??? I am still going to need to look forward and back beyond it. Yes, I am VERY familiar with full packet capture solution, since I helped build one.
So perhaps a compromise is:
1) we allow just a date *2015-11-23" without the T delimiter. This will make it SUPER easy for code to check for. This will allow people to give a date.
2) if you know more than just the date, maybe the hour, or maybe the hour and the minute, then things you do not know are zeroed out...
Thanks,
Bret Bret Jordan CISSPDirector of Security Architecture and Standards | Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
What if a programmatic event occurred at precisely "2015-11-24 00:00:00.000000" how would my logic discern the difference? My only issue here is with assuming zero values imply range of uncertainty. I am only arguing that there should be a mechanism
(optional?) to state uncertainty ***explicitly***. I'm 100% good with everything else here.
There are a number of Work Flow scenarios were this is relevant (User reports: "Oh yeah I saw an alert from the antivirus thingie sometime last week that warned me "ReallyBadStuff" will happen if I click OK to this, but I see these all the time and just
clicked OK like I always do". There are also legal implications for organizations in some communities for event reporting and time frames required under law to do so. In many cases you have to report before you can reasonably establish "when" something happened.
We need to be able to express "rough" findings rapidly to meet compliance regulation or contractual terms and then refine these dates and time as due diligence investigation occurs.
re: "I too want to know this... As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical. It does not have an effect in the actual work flow. "
It does have an effect on actual workflow for high Capability Maturity organizations that maintain strict device clock synchronization. Knowing which block of time to pull PCAPs from a Tape Library of PCAPs from a 10GBs channel for re=processing example
can make the difference of hours in processing time (in the case of Netwitness it takes 2 Hours/Terabyte to reprocess raw data packets through a Decoder back into Metadata and File Artifacts). Every hour of delay in recovering Actionable IOCs and Malware
Artifacts can result in 100's of additional APT compromised systems that need to be mitigated. So yes, most organizations do not maintain strict infrastructure Time Synchronization, but we should not presume that there is no value/requirement to support those
who have learned these hard lessons and have addressed same in their policies and procedures.
Folks - We are real close on this one. Is there a way to add Uncertainty as an optional parameter?
Patrick Maroney
President
Integrated Networking Technologies, Inc.
Office: (856)983-0001
Cell: (609)841-5104
I too want to know this... As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical. It does not have an effect in the actual work flow. If it does, please give some concrete real world work flow
examples of how this is going to help.
Thanks,
Bret
Bret Jordan CISSP
Director of Security Architecture and Standards | Office of the CTO
Blue Coat Systems
PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
HI Patrick - please see my earlier email around this topic, specifically the part around end-to-end workflow scenarios...
""Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday.
"
So lets assume for a moment that in the above use case the timestamp said "2015-11-24 00:00:00.000000" and
did not indicate any type of "precision" indicating "+/- 24 hours"... why do I as the consumer of that information care.. how does that affect my consumption of that threat report? How can I make more effective use of that information in my
system if I have that precision indicator? I am not going to do low-level temporal analysis on this type of indicator, so it doesn't matter from that perspective. What is the workflow where it does matter?
-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security |
www.securityintelligence.com
Without data, all you are is just another person with an opinion - Unknown
<graycol.gif>Patrick Maroney ---11/24/2015 04:17:59 PM---[+1] on use of syslog RFC and for support for Precision (in the literal sense). [+~] on being able t
From: Patrick Maroney <Pmaroney@Specere.org>
To: Terry MacDonald <terry@soltra.com>, "Barnum, Sean D." <sbarnum@mitre.org>, "Jordan, Bret"
<bret.jordan@bluecoat.com>, "Wunder, John A." <jwunder@mitre.org>
Cc: "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>
Date: 11/24/2015 04:17 PM
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000
Sent by: <cti-stix@lists.oasis-open.org>
[+1] on use of syslog RFC and for support for Precision (in the literal sense).
[+~] on being able to drive to consensus on this highly critical aspect "our thing" (even though we broke our priority list sequence)
But to Sean's point we do as a practical matter deal with "Uncertainty" (AKA "Precision") in Timestamps. There are a dozen use cases I can cite, but think I can simply use the "Well, were reporting this incident now as
required by Law, but we are only certain at this point that the activity occurred as far back as yesterday.
So we can clearly agree that there should be one and only one way of specifying temporal "values" (whether fixed, relative, or intervals). But his "one way of doing things" means we still need a way to convey "uncertainty"
where we need to communicate same.
Dose that makes sense?
Patrick Maroney
President
Integrated Networking Technologies, Inc.
Office: (856)983-0001
Cell: (609)841-5104
From: <cti-stix@lists.oasis-open.org>
on behalf of Terry MacDonald <terry@soltra.com>
Date: Tuesday, November 24, 2015 at 2:37 PM
To: Sean Barnum <sbarnum@mitre.org>,
Bret Jordan <bret.jordan@bluecoat.com>, John Wunder <jwunder@mitre.org>
Cc: "cti-stix@lists.oasis-open.org"
<cti-stix@lists.oasis-open.org>
Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000
I am fine with Trey’s proposal too. Specifically I support all 5 items on Bret’s email. I agree we do not have consensus on #5.
A few extra comments though, brought up by a review of the extra restrictions the Syslog
RFC (6.2.3 TIMESTAMP) places on implementations.
Is this an acceptable full list for the timestamp? (shamelessly borrowed bits from the syslog rfc).
The TIMESTAMP field is a formalized timestamp derived from [RFC3339].
Whereas [RFC3339] makes allowances for multiple syntaxes, this
document imposes further restrictions. The TIMESTAMP value MUST
follow these additional restrictions:
1.
An RFC3339 timestamp format MUST be used.
2.
All timestamps MUST be in UTC
3.
A timezone offset MUST NOT be used. All times MUST be recorded in UTC time.
4.
Usage of the "T" and "Z" characters are REQUIRED.
5.
The "T" and "Z" characters in this syntax MUST be upper case.
6.
The producer SHOULD include as much precision as its clock accuracy and performance permit.
7.
Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision
Examples:
nanoseconds = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
Note: The additional precision field is still up for debate as we currently do not have consensus.
What say everybody?
Cheers
Terry MacDonald
Senior STIX Subject Matter Expert
SOLTRA | An FS-ISAC and DTCC Company
+61 (407) 203 206 | terry@soltra.com
From:
cti-stix@lists.oasis-open.org [mailto:cti-stix@lists.oasis-open.org]
On Behalf Of Barnum, Sean D.
Sent: Wednesday, 25 November 2015 5:34 AM
To: Jordan, Bret <bret.jordan@bluecoat.com>; Wunder, John
A. <jwunder@mitre.org>
Cc: cti-stix@lists.oasis-open.org
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000
I am fine with #1-4
I disagree with #5.
Again, I will assert there IS a need for an explicit precision field. I have tried to explain the context a couple of time but apparently still have not done a good enough job.
Unfortunately, I do not have the cycles today to attempt to explain the scenarios for precision on this topic in any further detail.
Hopefully I will have a chance to do so tomorrow.
I just wanted to make it clear for now that we do not have consensus on #5.
Sean
From: "cti-stix@lists.oasis-open.org"
<cti-stix@lists.oasis-open.org> on behalf of "Jordan, Bret" <bret.jordan@bluecoat.com>
Date: Tuesday, November 24, 2015 at 1:24 PM
To: John Wunder <jwunder@mitre.org>
Cc: "cti-stix@lists.oasis-open.org"
<cti-stix@lists.oasis-open.org>
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000
So I think we are finally getting somewhere..... Before we claim that we agree, let me paraphrase and summarize just to make sure:
1) A timestamp format of yyyy-mm-dd-Thh:mm:ssZ MUST be used
Examples:
2015-11-23T13:35:12Z (for 1:35:12 in UTC format)
2) All timestamps MUST be in UTC a UI will change them as needed for an analyst
3) A timezone offset will NOT be used, all times will be recorded in UTC
4) Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision
Examples:
nanoseconds = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
etc.
5) There will be no extra precision field
Open questions
A) If the time of day is not known should it be:
i) zeroed out
Examples:
'2015-11-24T11:00:00Z' (known only to the 11th hour)
'2015-11-24T00:00:00Z (known only to the day)
ii) not included, just like the sub-second (if we do this, is this a violation of RFC3339)
Examples:
'2015-11-24T11Z' (known only to the 11th hour)
'2015-11-24TZ (known only to the day)
Thanks,
Bret
Bret Jordan CISSP
Director of Security Architecture and Standards | Office of the CTO
Blue Coat Systems
PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
On Nov 24, 2015, at 06:24, Wunder, John A. <jwunder@mitre.org> wrote:
I have a slight preference for Trey’s approach but would also be fine with Terry’s.
I prefer Trey’s because IMO support for a standard as-is is better than redefining the standard. RFC3339 allows for accuracy to any arbitrary (second) value and therefore the vast majority of date/time libraries will be able to support that. His approach allows
us to use the standard as-is. In fact, the requirement for millisecond precision may make things more difficult for those who just want to use RFC3339 processing libraries because they have to get those libraries to produce *our* version of RFC 3339 rather
than anything compatible with the standard.
I don’t think it will allow accuracy to the day unless we move back to ISO 8601 with all the other openness that it brings. But, I tend to agree with Terry that it will matter very little in almost every circumstance so we shouldn’t worry about it.
John
On Nov 24, 2015, at 5:28 AM, Trey Darley <trey@soltra.com> wrote:
On 24.11.2015 02:36:50, Terry MacDonald wrote:
I think that we have to put this whole discussion in perspective.
Most organizations have difficulty in discovering they have a
breach within days and weeks, not within a second. So going with
B) iii) and having the precision within 1 second realistically is
good enough in my opinion. It is far more likely that all the
clocks on the network are not synchronized, and all the tools are
reporting different and unrelated times and that they are way
more than a second out of alignment with each other. The zeroed
out millisecond timestamp doesn’t impact us much when we have
real-world problems such as that.
I agree with Terry however...
On 23.11.2015 20:51:09, Wunder, John A. wrote:
This is not to say that we need a precision field, just that if we
do it should be explicit rather than implicit.
...I also agree with John. On the one hand, I can't count the number
of times I've seen investigations get thrown under the bus due to
clock-sync issues. On the other hand, recent history has shown that
historical assumptions made about datetime can come back to bite us
with a vengeance.
So I think we should just use the standard explicitly. The producer
can use the level of precision they support and intend to communicate,
the consumer can easily recognize the level of precision coming from
the producer, and handle it appropriately. Doing this in the manner
illustrated below imposes less burden on the consumer than handling an
optional 'precision' field while accomplishing the same goal.
<snip>
#!/usr/bin/env python
import re
nanoseconds = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
seconds = '2015-11-24T09:42:54Z'
minutes = '2015-11-24T09:42Z'
# You get the idea...
timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes]
nano_re = re.compile(r'\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{9}Z')
micro_re = re.compile(r'\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{6}Z')
milli_re = re.compile(r'\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}Z')
sec_re = re.compile(r'\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z')
min_re = re.compile(r'\d{4}-\d{2}-\d{2}T\d{2}:\d{2}Z')
for ts in timestamps:
if nano_re.match(ts):
print('Found nanosecond time...')
elif micro_re.match(ts):
print('Found microsecond time...')
elif milli_re.match(ts):
print('Found millisecond time...')
elif sec_re.match(ts):
print('Found second time...')
elif min_re.match(ts):
print('Found minute time...')
</snip
There! Entirely explicit, uses RFC 3339 as intended, and there's one
clear way of doing things without resorting to optional fields.
--
Cheers,
Trey
--
Trey Darley
Senior Security Engineer
4DAA 0A88 34BC 27C9 FD2B A97E D3C6 5C74 0FB7 E430
Soltra | An FS-ISAC & DTCC Company
www.soltra.com
--
"It is always something." --RFC 1925
|