Hey all,
The developers at MITRE working on the validator and the schema put together the attached spreadsheet. It has a list of MUST and SHOULD requirements in the spec and whether they can be
tested in JSON Schema, python code, or untestable via automated processes.
The main caveat is that “not enforcing via automated processes” does not mean untestable. I reviewed the items marked NOT ENFORCING and IMO most or all of them are testable by looking at
products or content manually. There are a couple in gray areas (must not use tool to document malware, for example) but personally I don't see any problems with them. If anybody does have comments on specific items, let’s figure it out ASAP so we can move
forward with these specs.
Thanks,
John
PS: this is a good time to plug the validator and schemas that MITRE has created on behalf of DHS. As you can see from the spreadsheet, they’re able to test quite a bit of the STIX 2.0
requirements. You can find them in the OASIS Open repos:
https://github.com/oasis-open/cti-stix2-json-schemas and
https://github.com/oasis-open/cti-stix-validator.
From:
<cti@lists.oasis-open.org> on behalf of Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Date: Tuesday, November 22, 2016 at 1:58 PM
To: "Kemp, David P" <dpkemp@nsa.gov>
Cc: "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Subject: RE: [cti] Normative Statements
What makes this not testable is that "non-trusted destinations" are not defined by specification, and are thus up to the discretion of the consumer of the data.
As this is written, I as a consuming piece of software could decide that posting to twitter is a trusted destination, write my documentation as such, and thus anyone who sends me anything that is TLP:RED I can post on twitter. I would be meeting the specification,
as written - but obviously not as intended, because this requirement is not properly testable.
-
Jason Keirstead
STSM, Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com
Without data, all you are is just another person with an opinion - Unknown
"Kemp, David P" ---11/22/2016 01:22:43 PM---"Implementations
of TAXII servers that offer TLP MUST NOT forward STIX documents marked TLP Red to n
From: "Kemp, David P" <dpkemp@nsa.gov>
To: "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Date: 11/22/2016 01:22 PM
Subject: RE: [cti] Normative Statements
Sent by: <cti@lists.oasis-open.org>
"Implementations of TAXII servers that offer TLP MUST NOT forward STIX documents marked TLP Red to non-trusted destinations"
If this is the example requirement under discussion, it clearly appears to be testable, by human or machine, without psychic powers.
* IF a STIX document is marked TLP Red -- this has a yes or no answer. The overall marking of a document would be the high-water level of all its components' markings, and if no component is marked then the requirement does not apply. The requirement
could be met either by sanitizing the document by removing all offending components, or by not forwarding anything at all.
* TO non-trusted destinations -- this has a yes or no answer, assuming the TAXII server has a destination white list.
A counterexample would help understand why this might be regarded as untestable. "Source data" does not apply - if source data is marked but the STIX document is not, then the source processing system is at fault, not the TAXII server.
Dave
-----Original Message-----
From: cti@lists.oasis-open.org [mailto:cti@lists.oasis-open.org] On Behalf Of Peter F Brown
Sent: Tuesday, November 22, 2016 10:02 AM
To: Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Cc: Bret Jordan (CS) <Bret_Jordan@symantec.com>; duncan@sfractal.com; cti@lists.oasis-open.org
Subject: RE: [cti] Normative Statements
Well,
“So how can I say if they violated the spec or not, to know if the content is valid? How can an auditor know this?”
If the spec says “don’t pass x on” and you do, that would seem to be a clear – and testable – violation, no?
But I agree – it’s easier said than done. My main point is that several projects have gotten so bogged down in determining whether something is “testable” or “normative” that they actually miss the bigger prize of gaining buy-in to the use of the spec.
Cheers,
Peter