RE: [tosca] More on dangling requirements

[Chris Lauwers <lauwers@ubicity.com>]

Thanks Tal, these interactions are very helpful to get to the core of where weâre both coming from. More comments in blue 😊

 

On Tue, Nov 17, 2020 at 1:16 PM Chris Lauwers <lauwers@ubicity.com> wrote:

Yes, and presumably the Application node type already defines the âdataâ requirement to ârequireâ a node of type âDatabaseâ, in which case you donât even have to specify the âDatabaseâ type here. In addition, you will likely also want to specify a node filter to âfilterâ the set of Database nodes that will be considered to fulfill this particular requirement. The node filter (together with the node and capability types specified in the requirement definitions) define the âqueryâ that you will run against your inventory to find the set of suitable nodes for fulfilling this requirement.

 

How could a node filter satisfy the complex provisioning logic I detailed below, e.g. in the policy?

 

I think we continue to confuse âselectionâ (i.e. retrieving an existing node from an inventory) and âprovisioningâ (i.e. creating a new node). Dangling requirements are *ONLY* for retrieving a node from an inventory.

 

In the past we saw proposals for very complex branching-logic additions to node filters. Those would not only be very challenging to parse, but also they are very partial, because they only indeed apply to branching-logic. I don't see how we could ever create a grammar that would reflect how all orchestrators work. Some orchestrators would be able to do more (and we won't be able to express those in TOSCA) while other orchestrators do less and will have to be non-compliant.

 

Iâm not sure that creating a richer âqueryâ language for node selection based on logical operations (or, and, not) of (already existing) TOSCA constraint grammar is hard. Again, node filters not intended to specify âhow orchestrators workâ, theyâre just a mechanism for service designers to specify what types of resources are required by the various nodes created from their service templates.

 

 If the requirement is mandatory (i.e. the âoccurrencesâ keyword in the requirement definition has a lower bound that is greater than zero), then the orchestrator will find a suitable node during the requirement fulfillment phase, and create an edge between the node that has the dangling requirement and the node from inventory that was used to fulfill the requirement.

 

A single number is insufficiently simplistic. In any case, I am confused as to what you intend "occurrences" to mean. If you are agreeing that it has only to do with the instance model, but aren't we talking about runtime provisioning?

 

âoccurrencesâ are used as part of requirement definitions (i.e. at design time) to indicate how many times a specific requirement can (and needs to) be fulfilled. The occurrences keyword dictates/constraints how an instance model can be derived from a template. Do you think there is confusion about this?

 

You can absolutely do a âget_propertyâ to refer to the database. Since your template specifies that the target node for the âdataâ requirement is of type Database, your parser can validate that âvalidâ property values are being retrieved.

 

How would we do this? What is the entity name? I.e. { get_property: [ ???, ip_address ] }

 

You donât need to use the target node name in the TOSCA path _expression_. Using your âdataâ requirement from your previous example, you would use:

{ get_property: [ SELF, data, ip_address ] }

 

This uniquely identifies the âip addressâ property of the node that is used as the target of the âdataâ requirement of the node that âcallsâ the get_property function.

 

 

I donât think there is confusion either way: if a dangling requirement is not mandatory, it will not result in an edge in the instance model. If it is mandatory, it will result in an edge in the instance model (or the orchestration will fail if a suitable target node cannot be found).

 

Whether or not it's mandatory could be entirely up to the runtime environment. Moreover the exact number of occurrences might not be known during design time. And then how many "occurrences" are mandatory or optional or preferred might depend on runtime factors.

 

Wouldnât the service designer be in charge here? Could you give an example of where the runtime environment makes decisions about whether requirements need to be fulfilled or not?

 

TOSCA doesnât specify how the orchestrator is supposed to provide the (inventory) database, so Iâm not sure what additional flexibility is needed?

 

If not in TOSCA, how do you see this configured? How can I say that for a specific node in a specific service template I have specific (and conditionally complex) requirements?

 

I was just stating that TOSCA doesnât dictate whether the instance model inventory is maintained in a database, in the file system, in memory, or using some other mechanism. Of course TOSCA is used to define the node that needs to be retrieved from that instance model inventory.

 

The problem with policies as you use them is that they have absolutely no (language) semantics associated with them. All the semantics are encoded in the properties, which means that they can only be processed by an external domain-specific entity that knows what these properties mean.

 

That is absolutely correct, and is the core of our disagreement. I'm afraid that the dangling requirements dictates a runtime "selection" policy for orchestrators. I can assure you that it won't work for any of my use cases. If it ends up being a TOSCA feature, I will simply not implement it and will have be non-compliant.

 

Could you give examples of where runtime selection wonât work?

 

The idea of a "pure" TOSCA orchestrator is being taken too far here for me, because you seem to be dictating a very specific "selection" algorithm for all orchestrators. Would it really be impossible for you to have a "ubicity" profile which users can import and then gain access to specific behaviors of your orchestrator? I don't see how one size can fit all.

 

I think weâre confusing two different things here:

 

• A Ubicity profile would define domain-specific behavior that is defined using artifacts associated with Ubicity-specific node and relationship types. A pure TOSCA orchestrator would have no knowledge of the semantics encoded in these artifacts.
• However, a pure TOSCA orchestrator must absolutely be able to process all features expressed using the TOSCA language, including instantiation (i.e. creation of a service instance model from a service template), requirement fulfillment (i.e. finding matching nodes for mandatory dangling requirements in a service instance model) and substitution. Not only that, the behavior of these functions should be deterministic, in my opinion (i.e. different orchestrators should process them in an identical fashion).

 

 As I stated earlier, the âoccurrencesâ keyword in the requirement definition specifies whether the requirement is mandatory or optional. Do you  have example where âconditionally-optionalâ should be used?

 

Example: "Require databases if running on an x86 platform. Require two databases in redundancy mode if running with 16 GB RAM or higher. On all other platforms consider databases a nice-to-have soft requirement." Real world examples tend to be much more complex than this.

 

All of this can easily be handled using substitution mapping. You would create an abstract Application node that can be substituted by a variety of substituting templates. Using the substitution filter (which checks whether you have an x86 platform or not and how much memory you have), you select the appropriate substituting template. Each substituting template would then define its own (dangling) requirement for the database as appropriate.

 

And it can get more complex: perhaps later on in the runtime lifecycle (day 2) a database becomes available, and because it's a "nice to have" suddenly there would be a non-zero correspondence between the instance model and the world.

 

Iâm not sure I understand what you mean here.

 

Imagine that initially the cloud cluster does not have capacity for a database. But, a week later it is expanded and suddenly has capacity. So, now the node template can be instantiated. (The requirement has already been resolved in the design phase.)

 

I imagine that there would have to be some sort of âtriggering eventâ that causes the orchestrator to âre-calculateâ the topology which may then include a database. This relates to the âupdateâ and âupgradeâ scenarios we discussed almost a year ago.

 

If youâre talking about âselectionâ (i.e. requirement fulfillment i.e. find a node from inventory), then this is exactly what the âexpandedâ node filter syntax is for that we proposed several months ago. However, you use the word âprovisionâ in your example, which is different from âselectionâ. Assuming you âprovisionâ using substitution, then a substituting template for a virtual machine presumably will be different from a substituting template for a bare-metal server, and each substituting template will specify how much memory it needs.

 

"Selection" is simply too minimal a description for what happens. The process of turning a node template into something real (zero or more "things") will always go beyond what TOSCA grammar could provide.

 

Weâre not talking about turning a node template into something real. The TOSCA grammar only specifies how to turn a template into an instance model. How that instance model gets turned into âsomething realâ is up to the operation implementations, which using a âpure TOSCA orchestratorâ are provided using artifacts.

 

I think TOSCA currently supports all of this: a TOSCA orchestrator performs requirement fulfillment, or substitution. Both of these functions require âdecision logicâ to find the best node to fulfill a dangling requirement, or the best template to substitute an abstract node. If a TOSCA orchestrator wants to use AI/ML to help with this decision logic, it should feel free to do so.

 

OK, but how would you model this logic in TOSCA?

 

This doesnât need to be modeled. All TOSCA needs to do is specify the fact that there is a need for a node to fulfill a requirement, or a need for a template to substitute an abstract node. The decision logic for finding the best node or finding the best template is completely up to the orchestrator (and can use ML/AI if necessary) as long as the selected node or template satisfies the node or substitution filters defined using TOSCA.

 

However, based on your examples in this email, Iâm not sure we need to make any changes to the current âmodelâ for how a TOSCA orchestrator is expected to work.

 

Hard "no" from me on this.

 

Of course, I can do all the stuff I mentioned in the email already, and do. However, if TOSCA grammar allows for dangling requirements -- this is something that I will have to explicitly not support. My policy for "selection" looks nothing like node filters, and is indeed an area of constant innovation where I try new approaches and discover new orchestration challenges that require new thinking. E.g. I have one selection where I actually run a whole Ansible playbook just to gather the data needed to make the selection. I think it's not a good move for TOSCA to dictate orchestration behavior and will continue to argue against such attempts.

 

This may be the core of where our disagreement is. In my opinion, TOSCA provides a far superior orchestration paradigm than anything else in the industry (including Kubernetes) because it is declarative, intent-based, and model-driven (i.e. it maintains an instance model repository), and unlike Kubernetes it does this in a general-purpose domain-independent way.

 

Specifically, if youâre trying to fit TOSCA into an Ansible paradigm (where you donât have this instance model repository), then yes you may need to do âunnaturalâ things such as retrieving data (using a playbook) each time you need to fulfill a requirement, but then youâre giving up on many of the benefits that TOSCA has to offer. I think that rather than âdumbing downâ TOSCA to fit old orchestration paradigms such as Ansible, we should instead try to figure out how to use Ansible in the context of TOSCAâs superior orchestration paradigm and focus our efforts on continuing to improve the TOSCA language to best exploit TOSCAâs paradigm.

 

Thanks,

 

Chris