Re: [cti] Re: [EXT] [cti] [EXT] Re: [cti] TAXII Pagination Example Text

[Andras Iklody <andras.iklody@circl.lu>]

I have no idea how to make this work based on what you have said, but
perhaps I am too dumb to understand :)

[Tools supporting TAXII] <====TAXII====> Sensor collector <---our
format---> Internal tools/sensors.

If any external tool wants to query our sensor collector, it will not
want all data that it can retrieve thrown in its face in one shot.
Luckily, our internal tools/sensors might help us out here. However, I
don't see how we could do that in this case with the limitations that I
have described earlier. Again, maybe I've missed something.

Our sensor collector is not meant to store all the data. It is meant to
forward queries in this case to the tools behind it.

Best regards,
Andras

On 10.09.19 15:37, Bret Jordan wrote:
> I have tried to help you understand how this works in TAXII 2.1 and how
> you can make this work. Â
> 
> If you want to see something else, please write up a proposal. ÂPlease
> make sure to not break existing use cases with your proposal.Â
> 
> BretÂ
> 
> Sent from my Commodore 128D
> 
> PGP Fingerprint:Â63B4 FC53 680A 6B7D 1447 ÂF2C0 74F8 ACAE 7415 0050
> 
> On Sep 10, 2019, at 3:34 PM, Andras Iklody <andras.iklody@circl.lu
> <mailto:andras.iklody@circl.lu>> wrote:
> 
>> Not really. We can translate the pagination request and pass it on to
>> other internal tools, it doesn't have to be stored on the same tool that
>> has a taxii interface.
>>
>> Best regards,
>> Andras
>>
>> On 10.09.19 15:24, Bret Jordan wrote:
>>> If you are going to provide pagination of the data, then by very
>>> definition you will need to store the data. ÂAt this point it is no
>>> longer streamed, but stored data. ÂSo you are going to have to solve the
>>> problem of storing the data, even if that is for a short period of time.
>>> ÂBut it will still need to be stored.Â
>>>
>>> BretÂ
>>>
>>> Sent from my Commodore 128D
>>>
>>> PGP Fingerprint:Â63B4 FC53 680A 6B7D 1447 ÂF2C0 74F8 ACAE 7415 0050
>>>
>>> On Sep 10, 2019, at 3:16 PM, Andras Iklody <andras.iklody@circl.lu
>>> <mailto:andras.iklody@circl.lu>
>>> <mailto:andras.iklody@circl.lu>> wrote:
>>>
>>>> Again, writing a proposal is not something we will attempt again after
>>>> our previous attempts. We don't have the stamina or the will to go down
>>>> that route again.
>>>>
>>>> We are currently investigating whether it makes sense for us at all to
>>>> add TAXII connectors or not, hence I was asking how this is supposed to
>>>> work with data that is not stored in a local database but streamed.
>>>> Pagination would still make a lot of sense since we don't want to barf
>>>> back massive amounts of data due to that blowing the memory limits that
>>>> the devices we are targeting will have. (as reference, the other project
>>>> we're working on:
>>>> https://clicktime.symantec.com/3Ni5wRp5tgwXhTB5LhvhgyT7Vc?u=https%3A%2F%2Fwww.d4-project.org%2F)
>>>>
>>>> If I understand your answers correctly this is currently out of scope.
>>>> From the previous mails, Jason's explanation of why this is an issue was
>>>> much more concise and to the point than my poor attempt so I'll just
>>>> quote him:
>>>>
>>>> "I agree with the problem;
>>>>
>>>> The problem is rooted in the fact that assuming that a document has an
>>>> "insertion time", is assuming the document lives as-is in a database.
>>>>
>>>> This all goes back to the "STIX and TAXII are not a database" mantra."
>>>>
>>>> I hope that this is clear enough. If our issue is out of scope that's
>>>> fine too, again we're just looking for clarification, worst case it's
>>>> one fewer connector for us to implement :)
>>>>
>>>> Best regards,
>>>> Andras
>>>>
>>>>
>>>> On 10.09.19 15:01, Bret Jordan wrote:
>>>>> The current TAXII spec is addressing all of the use cases that have
>>>>> been
>>>>> brought forward. ÂIf you would like some additional functionality,
>>>>> please write up a proposal.
>>>>>
>>>>> In regards to what you are doing. You have said that need to write a
>>>>> solution to aggregate the data from multiple sensors. ÂSince you are
>>>>> writing the solution, you can write it how ever you need. ÂIf you need
>>>>> to paginate data from that solution you are writing, then by definition
>>>>> you will need to store that data for some period of time. ÂSo once
>>>>> again, since you are writing the solution and needing to store the data
>>>>> in a data store, you can easily solve this problem. Â
>>>>>
>>>>> If TAXII is missing support for some use cases, Âplease write up a
>>>>> proposal for these use cases and how you would propose solving
>>>>> them, and
>>>>> why current solutions do not meet those needs. ÂI think that would
>>>>> really help the technical committee understand what you need and why.Â
>>>>>
>>>>> BretÂ
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this mail list, you must leave the OASIS TC that
>> generates this mail. ÂFollow this link to all your TCs in OASIS at:
>> https://clicktime.symantec.com/381FEGvep8UnhzeE7GyP87V7Vc?u=https%3A%2F%2Fwww.oasis-open.org%2Fapps%2Forg%2Fworkgroup%2Fportal%2Fmy_workgroups.php
>>
>>