OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

cti message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Re: [cti] Normative Statements

If we write for implementers, then everything not testable needs a more
widened scope to be testable, otherwise the system itself has a design flaw (end-to-end) which will diminish the value of the whole endeavour.

Shouting does not always help, and that is what these RFC artificial
injected upper cased words are. If it ain't clear, either describe it in other words or indicate implementation defined behaviour.

Another tackle at these format vs. processing domain issues, is to define alternative behaviour expectations side by side - like profiles - in the main spec prose, and then construct projections as more and more strict conformance levels in the matching normative conformance section.

Like sample refactored instead of:
"""Implementations of TAXII servers that offer TLP MUST NOT forward STIX documents marked TLP Red to non-trusted destinations"""

Might become in main prose (white listing is easier to specify):
"""If TLP is offered, a TAXII server forwards STIX documents marked TLP Red to trusted destinations only (tag x)"""

and in conformance section for level so-and-so:
"""A conforming implementation MUST guarantee clause (tag x)"""

Or something like this.

Caveat - I do not know, if the latter approach would help here, just wanted to mention, that this may reduce the shouting and ease the reading without giving way for interoperability issues.

Being able to automatically extract normative statements by detecting shout-outs from a prose specification is always nice, as it eases testing for correctness and interoperability of implementations.

In my experience every real world spec, will need exemption rules and amending to the extraction process.

All the best,

Am 21.11.16 um 18:56 schrieb Bret Jordan (CS):

    Agreed... Too often we look at these documents through the lens of a
    developer and what a product can and can not do.


    *From:* cti@lists.oasis-open.org <cti@lists.oasis-open.org> on behalf of
    Peter F Brown <peter@peterfbrown.com>
    *Sent:* Monday, November 21, 2016 10:28:42 AM
    *To:* duncan@sfractal.com; cti@lists.oasis-open.org
    *Subject:* RE: [cti] Normative Statements


    Many TCs get wrapped around the axle on this.

    “Testing” compliance is not limited to testing some piece of running
    code, it can ask be adored as here to conforming with some procedural
    requirement (as often happens in many ISO standards, such as ISO 9000,
    27000, etc.



    Sent from my phone. Apologies for brevity, levity, and laxity: it’s hard
    to write on a moving planet

    *From: *duncan@sfractal.com <mailto:duncan@sfractal.com>
    *Sent: *Monday, 21 November 2016 17:05
    *To: *cti@lists.oasis-open.org <mailto:cti@lists.oasis-open.org>
    *Subject: *RE: [cti] Normative Statements

        "I would argue that if a normative statement can not be tested then it is

    not actually normative and is just a guideline."

        "MUST all normative statements be testable? "

    I disagree. Using the example below "Implementations of TAXII servers
    that offer TLP MUST NOT forward STIX documents marked TLP Red to
    non-trusted destinations". This is untestable BUT when the it does occur
    - you can say "you violated the spec". If it's non-normative, then it
    is not a violation if you do it. I vote normative wording if we require
    it, even it if not testable in all cases.

    Duncan Sparrell
    s-Fractal Consulting LLC
    iPhone, iTypo, iApologize

    -------- Original Message --------
    Subject: Re: [cti] Normative Statements
    From: "Wunder, John A." <jwunder@mitre.org <mailto:jwunder@mitre.org>>
    Date: Wed, November 16, 2016 8:47 am
    To: Jason Keirstead <Jason.Keirstead@ca.ibm.com
    <mailto:Jason.Keirstead@ca.ibm.com>>, Eric Burger
    <Eric.Burger@georgetown.edu <mailto:Eric.Burger@georgetown.edu>>
    Cc: "cti@lists.oasis-open.org <mailto:cti@lists.oasis-open.org>"
    <cti@lists.oasis-open.org <mailto:cti@lists.oasis-open.org>>

    As you guys are reviewing the documents can you be checking for
    this? I just looked through all of the MUST requirements across the
    documents and while there might be a couple ones in a gray area
    (it’s testable if you have the source data, but you can’t look at
    content absent the source data and validate it) but for the most
    part I think we’re in good shape.
    The SHOULD requirements are obviously a bit harder to evaluate and
    we could probably debate for years about them but if you see
    anything especially bad definitely bring it up.
    *From: *<cti@lists.oasis-open.org <mailto:cti@lists.oasis-open.org>>
    on behalf of Jason Keirstead <Jason.Keirstead@ca.ibm.com
    *Date: *Wednesday, November 16, 2016 at 8:07 AM
    *To: *Eric Burger <Eric.Burger@georgetown.edu
    *Cc: *"cti@lists.oasis-open.org <mailto:cti@lists.oasis-open.org>"
    <cti@lists.oasis-open.org <mailto:cti@lists.oasis-open.org>>
    *Subject: *Re: [cti] Normative Statements
    I would argue that if a normative statement can not be tested then
    it is not actually normative and is just a guideline.

    It should be noted that we aren't even talking about "automated
    testing" - the proposed normative statement is not even testable in
    the mind of a human reading the document, because they have no idea
    if the things in the bundle were intended by the producer to be
    related or not.

    As such, I agree with Alan that such statements serve little purpose
    in a spec and belong more in a set of implementor guidelines.

    Sent from my mobile device, please excuse any typos.
    Eric Burger --- [cti] Normative Statements ---

    "Eric Burger" <Eric.Burger@georgetown.edu

    cti@lists.oasis-open.org <mailto:cti@lists.oasis-open.org>

    Tue, Nov 15, 2016 7:28 PM

    [cti] Normative Statements

    MUST all normative statements be testable?
    I suppose it depends on what we mean by testable. A few have said
    that not everything that we have as normative statements are not, in
    fact, testable. I would offer that is a proof point of something
    that cannot be normative.
    Let us take an example:
    Implementations of TAXII servers that offer TLP MUST NOT forward
    STIX documents marked TLP Red to non-trusted destinations.
    This sounds like a fantastic requirement. However, this is what that
    requirement translates to when we write code:
    Implementations of TAXII servers that offer TLP MUST NOT forward
    STIX documents marked TLP Red to non-trusted destinations, unless
    they feel like it because it is impossible for the sender to know
    what the recipient does once they receive and decode the document.
    Now we can have such statements in requirements documents or system
    conformance documents. However, they are meaningless in protocol or
    document definition documents. In fact, I would offer they are
    dangerous. Let us consider this example. I am a consumer of CTI
    technology. I read the specs, and a TAXII server MUST NOT forward a
    STIX document marked TLP Red to non-trusted destinations. I am
    looking at a vendor, and their product “is fully compliant with the
    TAXII specification.” Too bad for me there is no way to hold them to
    the fire if they do improper forwarding. It’s way too late to call
    the Protocol Police.
    While I am on my soapbox, since I just saw a dialog here along the
    lines of “Bundles SHOULD not have related objects in them,” I would
    like to reiterate the best practice for MUST/SHOULD/MAY.

    * *MUST* is something that the implementation must do. If it is
    something the implementation must do, it should be possible to
    test for it, because if it is something it must do, one clearly
    can check to see if it does not do it.
    * *SHOULD* is something that the implementation MUST do, UNLESS
    there is an enumerated reason not to do it. That is the
    formulation for SHOULD: “The implementation SHOULD implement X,
    unless Y or Z are present.” [This highlights the Bundle
    argument: “Bundles SHOULD NOT have related objects, UNLESS they
    are related.” In English: the spec says NOTHING about the
    relatedness of objects in Bundles.] If you cannot enumerate when
    the SHOULD is not a MUST, then the SHOULD is a MAY.
    * *MAY* is something that might be nice, and if it is present,
    please don’t barf on it.

    Note that given the formulation of SHOULD, specifically that the
    conditions under which the implementation does not do the SHOULD,
    leads us to a clearer formulation of SHOULD, namely the conditional
    MUST. Using the above example, instead of:

    The implementation SHOULD implement X, unless Y or Z are present.

    It is clearer to say:

    If Y and Z are not present, the implementation MUST do X.

    Beating the dead horse: every SHOULD and MAY in the specification
    non-linearly increases the likelihood of implementation errors and
    interoperability failures. The spec is already extremely hard to
    implement - JSON does not eliminate cyclomatic complexity! There is
    no reason to hand our adversaries which is supposed to make or more
    resilient to attack an infrastructure begging for attack.

    --------------------------------------------------------------------- To
    unsubscribe from this mail list, you must leave the OASIS TC that
    generates this mail. Follow this link to all your TCs in OASIS at:

Agenda ( Tuesday 22/11  -  28/11 )
Go to calendar
calendar (1 of 1 enabled)

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]