RE: [cti-stix] STIX timestamps and ISO 8601:2000

["Jason Keirstead" <Jason.Keirstead@ca.ibm.com>]

Oh I see... I was actually always assuming people would be parsing at ingesting time.... storing the data in raw form and parsing at query time would never be workable at-scale in most software.

It depends on the scale of data being dealt with as to how expensive this can be at ingestion time. People should be thinking on the scale of potentially ingesting and parsing tens to hundreds of millions of items per second, at least that is where my mind is at. When dealing in numbers like this, any parsing load should be minimized.

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


"Struse, Richard" ---11/25/2015 04:59:22 PM---I don’t think I was clear. My point was that at point of ingest you could normalize any 8601 timest

From: "Struse, Richard" <Richard.Struse@HQ.DHS.GOV>
To: Jason Keirstead/CanEast/IBM@IBMCA
Cc: "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>, "Wunder, John A." <jwunder@mitre.org>
Date: 11/25/2015 04:59 PM
Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000
Sent by: <cti-stix@lists.oasis-open.org>

---

I don’t think I was clear.  My point was that at point of ingest you could normalize any 8601 timestamps to conform with your internal restrictions on timestamps and store those (normalized) 8601-compliant timestamps.  You’d have whatever overhead of dealing with the more flexible 8601 exactly once – at ingest.  After that your system could safely assume that all timestamps conformed to your internal requirements.  Not a perfect solution but it seems like a workable one.
 
From: Jason Keirstead [mailto:Jason.Keirstead@ca.ibm.com] 
Sent: Wednesday, November 25, 2015 12:16 PM
To: Struse, Richard
Cc: cti-stix@lists.oasis-open.org; Wunder, John A.
Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000

I don't see how this can be done on an implementation by implementation basis and also preserve cross compatibility.

If I want my system to be able to "ingest STIX", it needs to be able to parse and handle any timestamps that might be thrown at it. I can't make any assumptions as to what format they are in, *unless* the STIX specification requires that they are in a specific format - then I can make that assumption.

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


"Struse, Richard" ---11/25/2015 10:50:16 AM---Jason,

From: "Struse, Richard" <Richard.Struse@HQ.DHS.GOV>
To: Jason Keirstead/CanEast/IBM@IBMCA
Cc: "Wunder, John A." <jwunder@mitre.org>, "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>
Date: 11/25/2015 10:50 AM
Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000

---

Jason,

While I’m sympathetic to your point, isn’t this best addressed within a specific implementation?  For example, a system might normalize all incoming ISO 8601 datetime objects to an equivalent, but fixed subset of 8601 (e.g. adjusting to UTC, extending out with zeros any unspecified time fields, etc.)  That system could then safely assume that all STIX/CybOX datetime objects in its data stores would follow that form and lessen the processing overhead without adding a burden on other systems and users.  

Would that address your concerns?

Thanks,
Rich

From: cti-stix@lists.oasis-open.org [mailto:cti-stix@lists.oasis-open.org] On Behalf Of Jason Keirstead
Sent: Wednesday, November 25, 2015 8:43 AM
To: Struse, Richard
Cc: Wunder, John A.; cti-stix@lists.oasis-open.org
Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000

The one part of this I want to make sure people do not gloss over:

Do not make the mistake of confusing library support for reduced computational overhead. Regardless of if there is robust parsing support for a format in Java or Python or Ruby or whichever, said "parsing magic" can make the life of the programmer easier, but it does not reduce the overhead of that parsing. If the field can have N multiple different ISO 8601 compliant formats inside it then this potentially increases the computational burden by N for every new parsing attempt. Timestamp is going to be one of the most critical pieces of information for software to *always* parse from text into a number, so if we are not using EPOCH then the overhead should be as low as possible - This is why I am strongly pushing for a single format in one field and not "ISO 8601 with X or Y"

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


"Struse, Richard" ---11/24/2015 09:41:06 PM---Good points John!

From: "Struse, Richard" <Richard.Struse@HQ.DHS.GOV>
To: "Wunder, John A." <jwunder@mitre.org>, "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>
Date: 11/24/2015 09:41 PM
Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000
Sent by: <cti-stix@lists.oasis-open.org>

---

Good points John!

So, since there appears to be robust library support for manipulating ISO8601-formatted date/time values and since it allows the representation of reduced-precision values (such as a date + hour or date + hour + minute), is there any reason why we can’t agree on this and move on?

From: cti-stix@lists.oasis-open.org [mailto:cti-stix@lists.oasis-open.org] On Behalf Of Wunder, John A.
Sent: Tuesday, November 24, 2015 6:55 PM
To: cti-stix@lists.oasis-open.org
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000

All,

After thinking about this for a bit and seeing all of the back and forth (in particular Alex’s e-mail below), it occurs to me that maybe the right answer is just to use ISO8601 as-is. It’s a standard for a reason after all

Here are the advantages:

- ISO 8601 is an international standard. There are libraries in many languages that are able to parse it [1,2,3,4,5]. As Alex says, we’ll have better luck getting tools to adopt an RFC or ISO standard vs. getting them to use something we invent.
- ISO 8601 supports arbitrary precision. Most libraries will zero-out the precision, which is perfect for simple consumers: that’s exactly what we’ve discussed. If on the other hand your application cares about precision you may be able to find a library to do it (the advantage of an international standard) or you may have to do it yourself. But this is fine: simple things are simple, complicated things are more complicated.
- I tend to agree with Alex on timezones. As long as the timezone is specified it shouldn’t matter what we use. Let’s again just go with the existing standard and allow offsets or UTC (so long as some kind of timezone is specified).
- ISO 8601 also supports things like duration…for our purposes, we would want to limit it to actual timestamps.

To summarize: handling dates and times in programming is hard [6] and we could spend weeks trying to find the perfect solution. Let’s focus on cyber threat intelligence though and use the existing international standard for dates and times.

Also, in all of my decisions I think: is there an XKCD comic about this? Yes, there is: https://xkcd.com/1179/

John

[1]: http://joda-time.sourceforge.net/apidocs/org/joda/time/format/ISODateTimeFormat.html
[2]: http://stackoverflow.com/questions/127803/how-to-parse-an-iso-8601-formatted-date-in-python
[3]: https://github.com/arnau/ISO8601 (plus somewhat more limited support in standard libs)
[4]: http://stackoverflow.com/questions/1536633/parsing-iso-8601-string-to-datetime-in-net
[5]: https://developer.mozilla.org/en-US/docs/Web/_javascript_/Reference/Global_Objects/Date/parse
[6]: http://infiniteundo.com/post/25326999628/falsehoods-programmers-believe-about-time

On Nov 24, 2015, at 6:04 PM, Foley, Alexander - GIS <alexander.foley@bankofamerica.com> wrote:

Please do not mandate that times in STIX / TAXII must be in UTC. It seems to me that one of the benefits of ISO 8601 / RFC 3339 timestamps is including the time zone offset from UTC.

We have years of experience with this at $dayjob, where many logs that might eventually be parsed into STIX start from times represented in local time zones. In our experience, “defaulting” to UTC leads to all sorts of people / process / technology side effects, such as when a parser assumes UTC inputs and the person utilizing the parser does not know or realize to write the logic to convert local time into UTC.

I think we will have better luck getting tools to adopt an RFC or ISO standard for how they put times into logs (versus Unix timestamps, etc.) than to convince them to always log in UTC or to always output a time in any given format and then indicate a time zone elsewhere.

Thanks,

Alex

From: cti-stix@lists.oasis-open.org [mailto:cti-stix@lists.oasis-open.org] On Behalf Of Jordan, Bret
Sent: Friday, November 20, 2015 3:52 PM
To: Wunder, John A.
Cc: Terry MacDonald; cti-stix@lists.oasis-open.org
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000

I think I could come on board with that...

Thanks,

Bret



Bret Jordan CISSP
Director of Security Architecture and Standards | Office of the CTO
Blue Coat Systems
PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
On Nov 20, 2015, at 13:34, Wunder, John A. <jwunder@mitre.org> wrote:

Yes please.

A lot of languages will handle this automatically when parsing ISO 8601 timestamps so it probably isn’t a *huge* deal, but every step we take to make things more consistent is a positive in my book.

John
On Nov 20, 2015, at 2:52 PM, Terry MacDonald <terry@soltra.com> wrote:

Regarding timezone, I’ve always thought that we should be mandating all times should be UTC at the STIX/CybOX level. It should be up to the UI tools to display that UTC time in whatever timezone the viewer is in.

We always made our logging tools record in UTC timezone, as it meant that we could capture data from around the world and it would make sense.

Always having UTC makes it trivial when pulling together dates from many and varied places across the world. Rather than forcing specification of timezone, I would just like to mandate that the timezone will always be UTC, and that the tools that use STIX will do the translation when they present it to the user.

Cheers

Terry MacDonald
Senior STIX Subject Matter Expert
SOLTRA | An FS-ISAC and DTCC Company
+61 (407) 203 206 | terry@soltra.com


From: cti-stix@lists.oasis-open.org [mailto:cti-stix@lists.oasis-open.org] On Behalf Of Wunder, John A.
Sent: Saturday, 21 November 2015 1:07 AM
To: cti-stix@lists.oasis-open.org
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000

One of the changes we made in STIX 1.1 was the addition of a “precision” attribute next to all timestamps (except for @timestamp, which was intended to always be precise to the second or millisecond). The intent was to allow producers to convey times that were precise to a millisecond/second (possible in ISO 8601), day, month, and year (not possible in ISO 8601). The use case was incident and report timestamps where implying millisecond or even second precision was not accurate…the incident was reported on a given day and so the full timestamp would need to fake the time portion of the day.

To be honest I’m not sure that that use case is important enough to introduce the concept of precision, but we did explicitly add it in 1.1 so it’s something we should at least consider as an alternative to always requiring millisecond precision even in cases where’s it’s essentially faked.

The other thing STIX added on top of that is that timezone is “strongly suggested” (it’s optional in 8601). I’d be in favor of making it required unless any implementors have objections.

John
On Nov 20, 2015, at 8:57 AM, Jason Keirstead <Jason.Keirstead@CA.IBM.COM> wrote:

The problem is the "subset" potion. ISO 8601 actually does not specify how to represent milliseconds. So some people don't include it, some do, sometimes they also throw in nanoseconds...

I think we should extend ISO 8601 in the spec to include milliseconds and make it mandatory, so that it is well known exactly what the time format will be. Millisecond resolution will be required for many pieces of software operating with STIX.

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


<graycol.gif>Jerome Athias ---11/19/2015 11:48:25 PM---Back to the future ---------- Forwarded message ----------

From: Jerome Athias <athiasjerome@gmail.com>
To: cti-stix@lists.oasis-open.org
Date: 11/19/2015 11:48 PM
Subject: [cti-stix] Fwd: STIX timestamps and ISO 8601:2000
Sent by: <cti-stix@lists.oasis-open.org>


---





Back to the future


---------- Forwarded message ----------
From: Jerome Athias <athiasjerome@gmail.com>
Date: 2014-02-10 10:10 GMT+03:00
Subject: Re: STIX Content Revision and Revocation
To: "Wunder, John A." <jwunder@mitre.org>
Cc : Terry MacDonald <terry.macdonald@gmail.com>, Patrick Maroney
<Pmaroney@specere.org>, "Grobauer, Bernd"
<Bernd.Grobauer@siemens.com>, "Taylor, Marlon"
<Marlon.Taylor@hq.dhs.gov>, DisplayName <patrick.maroney@mac.com>,
"Barnum, Sean D." <sbarnum@mitre.org>, Kyle Maxwell
<krmaxwell@gmail.com>, Dave Dittrich <dittrich@u.washington.edu>,
stix-discussion-list Structured Threat Information _expression_/ST
<stix-discussion-list@lists.mitre.org>


I would see xs:datetime as "an implementation of strings formatted
according to a subset of ISO 8601:2000, documented in RFC 3339."
http://www.w3.org/TR/xmlschema-2/#isoformats
http://tools.ietf.org/html/rfc3339

so no "differences"

2014-02-10 Wunder, John A. <jwunder@mitre.org>:
> Hey guys,
>
>
>
> I have a quick question about RFC3339 vs. XML Schema's dateTime. I looked
> into them briefly and as far as I can tell they are very similar...some small
> differences in what each allows (capital T to denote time vs. either capital
> or lowercase, things like that). Are there important differences that make
> RFC3339 better than xs:dateTime that I'm missing? The nice thing about
> xs:dateTime is that, as long as STIX is in XML it natively validates, vs.
> RFC3339 which we would have to validate (and wouldn't work as well in
> programmatic bindings).
>
>
>
> Note: I'm basing this off of this mailing list post regarding Atom, so it's
> very possible I just don't understand the differences:
> http://www.imc.org/atom-syntax/mail-archive/msg13103.html.
>
>
>
> Thanks,
>
> John
>
>
>
> From: Terry MacDonald [mailto:terry.macdonald@gmail.com]
> Sent: Sunday, February 09, 2014 8:20 PM
> To: Patrick Maroney
> Cc: Grobauer, Bernd; Taylor, Marlon; DisplayName; Barnum, Sean D.; Wunder,
> John A.; Kyle Maxwell; Dave Dittrich; stix-discussion-list Structured Threat
> Information _expression_/ST
> Subject: Re: STIX Content Revision and Revocation
>
>
>
> [+1] again on the RFC3339 (In UTC with 6 digits of precision) for me too
> please.
>
>
>
> Cheers
>
>
>
> Terry MacDonald
>
>
> Terry MacDonald
>
>
>
> On 10 February 2014 14:02, Patrick Maroney <Pmaroney@specere.org> wrote:
>
> [+1] For a universal (optional) timestamp attribute( (RFC3339 in UTC with
> at last 6 digits of precision for 'time-secfrac'). Understand we will need
> to defer on other related attributes (like potential CRUD/change modality)
> to give it the attention this needs However, this change would help lay
> the foundations for a unified temporal reference (or at least the ability to
> assert same ;-).
>
>
>
> Patrick Maroney
> Executive Director
> Defense Security Information Exchange (DSIE)
> Office: (856)983-0001
> Cell: (609)841-5104
> pmaroney@specere.org
>
> ________________________________
>
> From: owner-stix-discussion-list@lists.mitre.org
> <owner-stix-discussion-list@lists.mitre.org> on behalf of Grobauer, Bernd
> <Bernd.Grobauer@siemens.com>
> Sent: Sunday, February 09, 2014 2:48:01 PM
> To: Taylor, Marlon; DisplayName; Barnum, Sean D.
>
>
> Cc: Wunder, John A.; Terry MacDonald; Kyle Maxwell; Dave Dittrich;
> stix-discussion-list Structured Threat Information _expression_/ST
>
> Subject: RE: STIX Content Revision and Revocation
>
>
>
> Hi,
>
>
>
> I pretty much agree with Marlon's answers to the open questions put by Sean.
> Looking at
>
> draft 2 of version 1.1 of Stix, I would like to add my vote to Marlon's
> answer to question
>
> 10 "Should an additional @timestamp attribute be added to each contstruct.".
> Marlon
>
> prefers the attribute, and so do I: timestamp information is such essential
> information,
>
> that it should be on top-level right next to the identifier rather than
> somewhere below
>
> in the (optional!!) "Information_Source/Time/Produced_Time" element.
>
>
>
> Also: I think on must be able to add timestamp information to really every
> object
>
> that has an identifier: I am not sure that every object I want to timestamp
> has the 'Information_Source'-substructure:
>
> making '@timestamp' an attribute next to any place where an '@id' attribute
> occurs makes sure
>
> that any object can be timestamped.
>
>
>
> So, in short: although this comes rather late in the timeline of STIX 1.1: I
> really think
>
> that draft 2 should be changed with regards to the location of where the
> timestamp information
>
> is kept: make '@timestamp' a sibling of '@id', '@id_ref' and
> '@timestamp_ref' rather
>
> than keeping the timestamp in "Information_Source/Time/Produced_Time".
>
>
>
> Kind regards,
>
>
>
> Bernd
>
>
>
>

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that
generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php

---

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.