Re: [tosca] Agenda for Tuesday Jan. 19 2021 TOSCA Language Ad-Hoc Meeting

[Tal Liron <tliron@redhat.com>]

See attached my reimagination of the first slide.

On Mon, Jan 25, 2021 at 5:09 PM Chris Lauwers <lauwers@ubicity.com> wrote:

Hi Tal, comments in-line

Â

From: Tal Liron <tliron@redhat.com>
Sent: Wednesday, January 20, 2021 12:04 PM
To: Chris Lauwers <lauwers@ubicity.com>
Cc: tosca@lists.oasis-open.org
Subject: Re: [tosca] Agenda for Tuesday Jan. 19 2021 TOSCA Language Ad-Hoc Meeting

Â

Hi all, I will do my best to offer clear comments! I have decided to stick to the first slide only, in order to avoid making this email too long.

Â

The dotted line assumes that the "service instance model" is in the scope of the TOSCA processor, and that in turn is the input for the orchestrator. This introduces two problems:

Â

1. I think we need to be careful to not confuse âfunctional architectureâ with âsoftware architectureâ. Slide 1 only tries to define a minimal âfunctional architectureâ, i.e. a diagram that lists the minimal set of âfunctionsâ that need to be performed by *ANY* piece of software that claims to be able to process TOSCA files in order to deploy services. It doesnât try to specify how these functions must be implemented or how they are grouped together. It just tries to define the minimal set of functions that *must* exist.
2. Based on our earlier conversations, I believe you agree that a service defined using TOSCA must be *COMPLETE* before it can be orchestrated without errors. For TOSCA services, being complete means (at a minimum) the following:

1. All defined properties must have actual values. This means that all intrinsic functions (such as get_input or get_property) must be evaluated (which implies that a complete set of the required input values must have been provided).
2. All mandatory requirements must have a target node in the topology graph (or, said a different way, the topology graph must not have dangling requirements)
3. A substituting topology must exist for all nodes that are marked for substitution (using the âsubstituteâ directive).

3. There are a number of ways to ensure that a service defined using TOSCA is complete:

1. In the extreme case, service designers can define TOSCA topology templates that are entirely complete. This means that these topology templates have hardcoded property values (that donât use intrinsic functions), that the topology template specifies target node templates for all mandatory requirements in all node templates, and that the topology template does not mark any node templates for substitution. While this could be done, I assume we all agree that there is not a lot of value in using TOSCA this way.
2. The more flexible way is to defer âcompletionâ of the TOSCA topology until deployment (âorchestrationâ) time. This means that a piece of software âprocessesâ a TOSCA topology template with the goal of making it complete. This may include evaluating all functions, fulfilling all mandatory dangling requirements, and creating substituting topologies for all nodes marked for substitution.

4. Assuming youâre not objecting to the need for deferring the âcompletionâ of topology templates until deployment time, then everything else seems to be just disagreements about terminology. Here is the terminology I proposed in the slides:

1. TOSCA Processing (or âTOSCA Processorâ): the function that takes a TOSCA topology template and âcompletesâ it so it can be deployed (âorchestratedâ)
2. TOSCA Orchestrator: the function that takes a completed TOSCA topology template and deploys a service on external resources in the âreal worldâ
3. Instance Model: a fully completed TOSCA topology (based on a corresponding TOSCA topology template).

Â

Iâm open to any and all names that make sense, but I hope we can all agree that what I outlined above is the absolute minimal description of what needs to be done by any piece of software that deploys TOSCA services. Saying that we canât support the minimal functionality listed above would be like saying we need to support C++ but we canât require a compiler or a linker ð

Â

1) Orchestrators may very well come with their own way to manage deployable services. Flexible model-driven orchestrators may be able to accommodate TOSCA models directly, or at least a subset of supportabe features. In such cases the TOSCA processor might not be an external component but rather integrated into the orchestrator to work directly with its native model system.

Â

I completely agree, but youâre making a software architecture argument rather than a functional architecture argument. If your âorchestrator softwareâ knows how to âprocessâ TOSCA models directly, then by implication it includes a âTOSCA Processorâ function. Itâs perfectly fine (and very much desired) for that TOSCA Processing function to work directly with the ânative model systemâ used by the Orchestrator software, but that doesnât take away from the fact that the âTOSCA Processingâ function results in a fully completed model somewhere.

Â

The concept of "service instance model" is still grammatically relevant to TOSCA, but the dotted line in such cases would leave the "service instance model" out, as it would actually be part of the orchestrator.

Â

What we call the âservice instance modelâ is nothing more than a fully completed TOSCA topology that was created from a TOSCA topology template. Not only is this concept âgrammatically relevantâ to TOSCA, it is an absolute must-have for many TOSCA features such as node filters, condition clauses, intrinsic functions, and more. The dotted lines are not intended to imply which piece of software is expected to âimplementâ the âservice instance modelâ but rather which functions are expected to access it. We can put it outside of any dotted lines if that makes you more comfortable.

Â

Another way to look at this is that there is slippage between "grammatical architecture" and "functional architecture", which is the title of this slide deck.

Â

Iâm not sure what you mean by âgrammatical architectureâ

Â

2) As Thinh pointed out, this also introduces a standards compliance challenge, which will have cascading effects for other standards that refer to TOSCA. I think it's most responsible for us as a standards body to allow for TOSCA processors that do not introduce an intermediary state, which indeed has been the norm for many existing implementations. Again, I absolutely agree that the concept is very important to clarify the functionality of intrinsic functions, relationships, substitution, and other features, but we need to be careful not to assume how such functionality would be implemented. E.g. Puccini implements intrinsic functions against the intermediary state, but it's easy to imagine other implementations that rely on metadata stored in controllers or the cloud platform itself.

Â

I donât think such implementations are easy to imagine at all. For one, you need the fully completed model (your âintermediate stateâ) to decide which controllers or cloud platforms need to be used, and you may need to create entities on the cloud platform that can act as the âcontainersâ for the metadata you have in mind. Those cloud platform entities would need to be created from the very metadate that define the âintermediate stateâ. This is a chicken-and-egg problem that cannot be solved. I canât imagine any real TOSCA implementation where the TOSCA software doesnât keep its own data structures to represent the model (your âintermediate stateâ) from which services must be deployed.

Â

And, as politely noted in the slides, I am uncomfortable with calling it an "instance model" and indeed boxing up "instantiation" as a processing component. This is related to the problems mentioned above: for implementations that do not use an intermediary state there would be no such "instantiation", at least not as diagrammed here. "Instantiation" would be part and parcel of the deployment process itself. Using my example above: it's only when resources are deployed that you can inject metadata. If anything, such "instantiation" as we are referring to here might even happen after the resources are created.

Â

Youâre pointing out exactly the âchicken-and-eggâ problem that illustrates why âthe fully completed topologyâ is an absolute must-have before you can deploy. There cannot be a âdeployment processâ is there is nothing from which to deploy.

Â

Furthermore, it might not even be a phase per se. In the two implementations of a TOSCA processor that I've worked on, very different approaches were used. In Puccini I have indeed implemented this as a phase, the result being "normalized" data structures. These "normal" structures are "flattened" versions of the template entities (node templates, policies, groups) with the entire type hierarchy and inheritance having been validated and resolved, e.g. all properties are listed, default values assigned if necessary, etc. Importantly, intrinsic functions are not called in this phase, but rather normalized (and validated) call stubs are inserted instead. In the ARIA project we used a very different approach: everything was handled by lazy evaluation, e.g. when you tried to read the value of a node template's property it would reach through the various hierarchies, validate/inherit on-the-go, and also call intrinsic functions. There was thus no specific normalization phase. Having worked with both approaches, my opinion is that a separate phase is preferrable. However, this is an implementation detail that we should not enforce.

Â

Both of those are OK, but they are implementation details. In the end, all functions need to be evaluated (lazily or otherwise) to create a âcompleteâ representation from which a service can be orchestrated. Again, weâre not talking about âphasesâ in a software implementations, weâre talking about functions in a functional architecture that *must* be provided somehow.

Â

Thinking through these problems, my current feeling is that it may be best to abandon trying to standardize the functional architecture and instead focus on the grammatical architecture. My proposal is that in the text of the spec we refer, where necessary, to the "normalized representations" of node templates, without making assumptions for how these are generated or whether they are stored in an intermediary state, the platform itself, or anywhere else. This is actually a non-trivial and powerful requirement, because it assumes that TOSCA processors would need to have access to such normalized representations. Until TOSCA 2.0 this could not be assumed.

Â

Iâm still not sure what you mean by âgrammatical architectureâ, but what youâre describing is exactly the âfunctional architectureâ weâre trying to define. This functional architecture is absolutely necessary to provide TOSCA users with a âmental modelâ or âoperational modelâ that explains how they can expect their TOSCA templates to turned into deployed services. And yes, this includes defining a requirement for the ânormalized representationsâ of TOSCA topologies (for which weâve been using the term âservice instance modelâ in previous discussions).

Â

The concept of a "normalized representation" goes a long way to clarifying how various intrinsic functions work, but most importantly in my view it explains TOSCA relationships. The satisfaction of requirements, and thus the construction of the graph, happens between the normalized representations. This will allow us to think more clearly about what "occurrences" could mean for capabilities and requirements.

Â

Completely agree, and this confirms my suspicion that weâve mostly been arguing about terminology rather than fundamental concepts.

Â

I can already guess that there would be objections to my proposal, because some of you would very much like to require that intermediary state to exist in order to enable controversial features such as "dangling requirements" and mulitplicity of template "instances", which would require a clear notion of an instantiation phase. And indeed this is one reason why I object to such features.

Â

Once we agree on whether to use the term ânormalized representationâ or âservice instance modelâ, then we can go back to having discussions about specific features in the language, such as âdangling requirementsâ and âcardinalityâ. I hope we can use this weekâs meeting to achieve consensus, and move on to the fun stuff ð

Â

Â

Thanks,

Â

Chris

Attachment: TOSCA Functional Architectures.pptx
Description: application/vnd.openxmlformats-officedocument.presentationml.presentation