Re: Data Markings / Marking Definition Redaction patterns

[Stephen Russett <stephen@digitalstate.ca>]

Hi All

Wanted to share a update for the conversation around redaction.

I have implemented a working Object Marking and Granular Redaction Process for all Bundleable objects (SROs, SDOs, Marking Defs). Redaction works for each object as well as when generating a bundle.

So to try and spur some conversation here:

Consider the following:

You have a TLP Object for a Attack Pattern as a Granular Marking:

Tlp tlp = Tlp.builder().tlp("red").build()
MarkingDefinition markingDefinition = MarkingDefinition.builder()
        .definition(tlp)
        .definitionType("tlp")
        .build()
GranularMarking granularMarking = GranularMarking.builder()
        .markingRef(markingDefinition)
        .addSelectors("granular_markings", "created_by_ref")
        .addSelectors("created")
        .build()
AttackPattern attackPattern = AttackPattern.builder()
        .name("some Attack Pattern")
        .addGranularMarkings(granularMarking)
        .createdByRef(Identity.builder()
            .name("some Identity")
            .identityClass("individual")
            .build())
        .build()

Notice the 3 selectors.

Now on the STIX spec side we have the @Redactable annotation:

something like:

@JsonProperty("created_by_ref") @JsonInclude(value = NON_EMPTY, content = NON_EMPTY)
@JsonIdentityInfo(generator= ObjectIdGenerators.PropertyGenerator.class, property="id")
@JsonIdentityReference(alwaysAsId=true)
@JsonDeserialize(converter = DomainObjectOptionalConverter.class)
@Redactable(useMask = true, redactionMask = "identity--__REDACTED__")
Optional<IdentitySdo> getCreatedByRef();

@NotNull
@JsonSerialize(using = InstantSerializer.class)
@JsonFormat(shape=JsonFormat.Shape.STRING, pattern = StixDataFormats.TIMESTAMP_PATTERN, timezone = "UTC")
@JsonProperty("created")
@Value.Default
@Redactable(useMask = true)
default Instant getCreated(){
    return Instant.now();
}

@NotNull
@JsonProperty("granular_markings") @JsonInclude(NON_EMPTY)
@Redactable
Set<GranularMarkingDm> getGranularMarkings();

Notice the three variations of configuration on the @Redactable annotation

@Redactable(useMask = true, redactionMask = "identity--__REDACTED__")

@Redactable(useMask = true)

@Redactable

and now when we generate a Bundle the output of the JSON would be:

{
  "type": "bundle",
  "id": "bundle--3d6bdcdd-2137-4e97-a8a4-8020dd30bc8d",
  "spec_version": "2.0",
  "objects": [
    {
      "type": "attack-pattern",
      "id": "attack-pattern--0f4d3058-f4de-4743-ae4c-988645309d92",
      "created_by_ref": "identity--__REDACTED__",
      "created": "ââREDACTEDââ",
      "modified": "2018-12-19T20:49:06.403Z",
      "revoked": false,
      "name": "some Attack Pattern"
    }
  ]
}

Notice the granular_markings property has been removed completely, as the configuration of @Redacted did not state to use the mask.

A Object Marking works similar but will redact the entire document. So if you generate a Bundle wth the attack pattern and it has a non-passing object marking then you would get something like:

{
  "type": "bundle",
  "id": "bundle--3d6bdcdd-2137-4e97-a8a4-8020dd30bc8d",
  "spec_version": "2.0",
  "objects": []
}

The way this all works is through the concepts of âSubjectsâ. A Subject has âObject markingsâ (Statements and TLPs) (for now only TLPs have enforcement logic). and the Subject must have all TLPs stated during JSON generation or else redaction kicks in. In the above example the Subject only had the White TLP but was missing the Red TLP. This is obviously a simplified example, and more complex Object Markings need to be adapted for your specific use case in the processor.

configurable logic is added throughout the processor for things like âIf a Granular Marking selector is for a non-existing property, then it will throw a error and failâ. But could also just skip it and log that is skipped it. Currently its setup for Fail fast under the premise.

I have taken the following approaches:

1. Required Fields have Masked Redactions

2. Non-Required Fields have Removal Redactions

The redaction schemes i showed above are just working samples and are not necessarily ârecommended patternsâ.

Further considerations that could be patterned as things like:

- When redacting nested arrays (such as labels, or external references), should a redacted object (or even a string) be a {} / "" Âor just completely removed? There are use-cases for both. (such as being allowed to know the âcountâ but not see the actual items inside). I opened this issue today:Âhttps://github.com/oasis-tcs/cti-stix2/issues/125Âwhich is a example where additional redaction logic could be passed around allowing downstream consumers to know proper redaction protocols when sharing.

- Object Markings (such as statements and TLPs) can have additional x_ fields providing additional logic gates for redaction. Thus allowing âIf being shared with XYZ identity, or a class of Identity or industry or under specific ABC circumstances, then specific redactions need to be applied as be the markings.

Very interested to get feedback from the group on usage and types of redactions scenarios listed above and in my previous email.

Thanks!!

Steve

From:ÂStephen Russett <stephen@digitalstate.ca>
Reply:ÂStephen Russett <stephen@digitalstate.ca>
Date:ÂDecember 17, 2018 at 4:21:10 PM
To:Âcti-users@lists.oasis-open.org <cti-users@lists.oasis-open.org>
Subject:Â Data Markings / Marking Definition Redaction patterns

Hi all

Looking to get some community feedback on common use cases for redaction patterns being used for Data Markings.

As I see the spec, there are two overall considerations:

1. Object redactions

2. Field level redactions

For #1, what are the most common needs for users? Complete removal/non-existant knowledge? or returning empty objects (if a array had 5 redacted items then there would be 5 empty objects.

For #2, where selectors are defining redactions, what are the common needs? Field Removal, Field masking (*****), common replacements like â__REDACTED__â.

And what about when fields are redacted that are a âRequiredâ property for that object? What are the common use cases and problems that you have run into for the consumer to receive non-spec match json (incorrect values and/or missing required fields)?

I am currently implementing the Java lib and want to quickly configure these various scenarios:

current prototype is as follows:

public interface SomeSDO extends ....... {

    @NotBlank
    @JsonProperty(âsome_property")
    @Redacted(processor=SomeRedactionProcessor.class, mask="__REDACTED__")
    String getSomeProperty();
...

Where @Redacted will be a configurable annotation. ÂBasically you can globally, per class/object type and per property. Where each is about define the redaction schemes for Object Marking Refs and Granular Refs. Â

The way this works is during serialization, the processor looks at each property and determines if redaction is defined. ÂIf so, then it reprocesses that property based on the redaction scheme. Before all this occurs, the call for json serialization is given a input of Bundleable objects (SDO, SRO, Marking Definitions) that are used during evaluation of the redaction. Âbasically this is a comparison of the âsubjectâ (the consumer of the json) and the classifications applied at the producer side of the json; anywhere the subject is missing the required Object Markings, the redaction processor clean the data as defined by the processing rules.

The usage here is that the producer will have defined âfiltersâ that are groups of Marking Objects that are the âWhat Marking Objects the Consumer will haveâ, and the JSON is generated based on the rules that are passed and not passed. (of course lots of different variations can be applied as well).

It would be great if some of the community can share the common scenarios they have been seeing and hopefully we can adjust the redaction based on this!

Thanks!

Steve