RE: Differentiating TOSCA from HEAT

["Bruun, Peter Michael (CMS RnD Orchestration)" <peter-michael.bruun@hpe.com>]

Hi Chris,

 

Again good questions.

 

And what do you mean by âproduction rulesâ

Each line in a BNF defines how to produce some part of the language. In general production rules are also a key part of logic reasoning â producing true results from other true results.

Graph grammars have similar rules for transforming one valid graph into another. This is particularly relevant for day-2 scenarios where you want to transform an already deployed topology into something modified to express a changed intent, for example.

 

Substitution mappings are also a kind of production rules in this sense.

 

BNF just formalizes syntax, not the language and its associated semantics.

 

Of course a grammar is just syntax, a full language consists of grammar, static and dynamic semantics. So the âGraph Grammarâ is not the DSD language â it is just the structural backbone to which the semantics are attached.

 

The grammar of TOSCA allows you to define any topology consisting of nodes and relationships, i.e. any graph. Now YAML is a bit tricky to express using BNF, but in principle, we can make a BNF grammar that defines all valid TOSCA specifications (I still think we should attempt that).

 

Since every conceivable concept in the universe can be described as a graph, this makes TOSCA infinitely flexible, but it also means that out-of-the-box it provides no information and no constraints whatsoever on which topologies are meaningful and which ones are not. As I wrote below: Flexibility is not the same as good. So for a given class of problems â topologies that realize customer access, VPN, etc â it is useful to add information to describe which topology graphs are meaningful realizations of that intent, and which ones are not. There is an elaborate example of this in the presentation I originally gave to the TC.

 

You can see the graph grammar as a âWizardâ that helps you define and evolve over time those topology graphs that realize a certain class of intents. Adding a site to an access topology makes sense, but adding a CRM system does not. This is very similar to âsyntax directed editingâ realized by many IDEs for programming languages â it uses a grammar to define how to expand, within a set of rules and policies, an existing definition to another one.

 

So where a TOSCA specification allows you to define one topology, a specification in the DSD language corresponds to a grammar for a class of TOSCA topologies. Yes, TOSCA is more than a single topology, because substitutions do actually also define âproduction rulesâ for expanding a grammar, even recursively, as you point out. So in that sense, TOSCA with substitutions is also defining a graph grammar. So you are actually familiar with the concept â but it can be greatly simplified.

 

It is quite simple to translate TOSCA to DSD â we do have such a translator, so this is very similar to what Tal is doing when translating TOSCA directly to K8, and so I very much support that approach.

 

The difference is that DSD takes on the task of being a grammar of topology-graphs more directly. Interestingly, the requirements-capabilities functionality (including dangling requirements) is expressed in DSD as a very simple, easy to understand function call (as in any other function you could think of in TOSCA or HOT). It is extremely simple to read and understand, making the complexities of understanding substitution mappings in TOSCA a bit odd for me to listen to.

 

what do you mean by âa BNF grammar is not fully compositionalâ

Ok, I may be nit-picking a bit. You can only add a rule in a BNF by adding the new production into one or more existing rules â so you need to modify something in what is already there. Think of object orientation: In Java you can add a class saying âY extends Xâ, instead of having to edit âXâ to add âYâ there. Martin Fowler has some other great refactoring examples using virtual methods: Assume you have an enum class with values A, B, and C. You could write your code so that every place the enum is used, you have a switch-statement with a case for A, B, and C. But what if you later add D to the enum? You then have to go visit all those switch-statement to add a D-case branch. This is not compositional. But if instead you define semantic abstract methods on the enum, and use those everywhere, then when you add D into the enum, you just define those metods for D, and the rest of the code will now work with D without modification.

So when I say âcompositionalâ, in this context I basically mean that I can add things.

 

Again, req-caps have that property in TOSCA â you can add something new with some capabilities, and then it may be picked up when matched by some existing requirements.

 

Notice, that I am fond of TOSCA â it has all the right elements. Only sometimes, things get a bit more complicated than necessary. Why do we have 6 (or is it 7?) different inheritance hierarchies in TOSCA â I have just two in DSD, and that nicely covers everything I see in TOSCA. It just makes life hard for users to understand more concepts than are really needed.

 

I assume we agree that TOSCA is denotational as well?

Definitely.

 

Doesnât this example mix and match the semantics of the language vs. the semantics of the program created using the language?

Hmm. There is a coupling in that a language feature can cause problems with hard-to-understand language usage. The comefrom is an example of that.

You should be able to inspect each piece of code written in a language to understand what it does without having to understand all other parts of the program. And once you understand what it means it shouldnât change if you modify some other part of the program. An example that violates this is macros in C. A program that makes heavy use of macros can be extremely hard to understand because adding a new macro could have unforeseeable consequences.

Yes, the individual program is doing the misuse of macros, but it is the language feature that makes this likely to happen.

 

This is important for language design â particularly for me in DSD, but I am not saying that TOSCA has any such a problem â this is really just my own concern as a language designer.

 

I absolutely try to look at TOSCA as a programming language. My orchestrator doesnât know how to do anything other than what it is programmed to do by TOSCA service templates, i.e. by applying TOSCA language semantics to those templates. Also, substitution mapping is definitely intended to be recursive.

 

You could define a programming language using XML or YAML syntax. There are rules for XML and YAML, but they have no semantics of their own â it is all down to how they are interpreted. Having a system that interprets XML or YAML as a programming language does not turn XML or YAML into programming languages themselves.

 

Now TOSCA is a lot stronger and has more semantics that XML or YAML, but it still fits very precisely into the OASIS family â I mean this in a very positive way.

 

To be a âprogramming languageâ TOSCA needs to be Turing-complete. Function and Logic programming languages achieve Turing-completeness through recursion. I guess you could achieve Turing completeness in TOSCA using recursively expanding substitution mappings, but the purpose of substitution mappings is not to make TOSCA Turing-complete. TOSCA is not striving to be a programming language in this sense, and it shouldnât.

 

Basically, I find the semantics of TOSCA very loosely defined. Yes, there are the things you mention, but there is a lot of room for interpretation. If you give the same TOSCA definition to your orchestrator, to one of Talâs and to mine, the result would not be expected to be the same, would it?

 

As I remember our discussions over the past year, we have explicitly avoided any such strong portability requirements for TOSCA. This is fine â it is a deliberate decision, and it is ok. We just need to distinguish between the specific ways that each of us interprets and strengthens TOSCA in our implementations and TOSCA itself.

 

I donât look at âundeployâ as a way to âundoâ

Ok.

 

I maintain that there really are only 5 âcanonicalâ entry points into a TOSCA service template to which all other actions can be reduced: deploy, undeploy, update/modify, upgrade, and ârun workflowâ

I recognize these, but I also have ârecreateâ. Assume the topology you deployed in the infrastructure is somehow damaged out-there in the real world, then recreating it or part of it is really a useful action to recover.

Basically, I find two modalities out-there in the target systems. Set or Recreate. Some things can just be updated by sending a new declarative update. In other cases what was there before needs to be destroyed before something new can be pushed. Note that ârecreateâ does not mean just to recreate in exactly the same way â but rather to recreate under new circumstances. In a TOSCA context, an entry point to reevaluate the conditions for a previously performed substitution may lead to a different substitution happening than the last time. So in DSD that would be an example of ârecreateâ. Maybe that is what you mean by âupgradeâ?

 

Cloud native orchestration

I think we should wait for Talâs input. I am no expert in that. I just gave it a shot.

 

Peter

 

From: Chris Lauwers [mailto:lauwers@ubicity.com]
Sent: 8. januar 2022 00:58
To: Bruun, Peter Michael (CMS RnD Orchestration) <peter-michael.bruun@hpe.com>; Tal Liron <tliron@redhat.com>
Cc: tosca@lists.oasis-open.org
Subject: RE: Differentiating TOSCA from HEAT

 

Hi Peter, more questions (that will show my ignorance) 😊

 

Great questions. There is a limit to what more, I can disclose. Looking up patents with my name can give some information.

 

-          Could you please explain what you mean by âcompositional graph grammarâ vs. âjust a graphâ?

 

A Graph Grammar corresponds to a Topology, like a BNF grammar corresponds to a program written in the language defined by that grammar.

 

These donât sound to me like different types of the same âclassâ of things. For example, wouldnât you be able to create a BNF grammar for your âGraph Grammarâ. BNF just formalizes syntax, not the language and its associated semantics. If I understand correctly, wouldnât the differences be more in the types of data supported and in the operations supported on those data? Traditional languages operate on strings, numbers, objects, etc. whereas I assume a graph language operates on graphs?

 

A BNF grammar for a programming language defines a class of texts that are syntactically valid for that language. You add a static semantics (type-checking, etc. to further constrain valid programs). A BNF grammar is not fully compositional, because you cannot add new production rules without modifying existing rules, but the BNF language itself could easily be modified to become fully compositional.

Apologies for being a little dense again, but what do you mean by âa BNF grammar is not fully compositionalâ. What is it that canât be composed? And what do you mean by âproduction rulesâ? What is it that is being produced (for which rules are needed)?

 

Furthermore, you usually want the language to have a denotational semantics â adding a production to the graph should not alter the semantics of existing constructs. The DSD language is compositional â you can incrementally add new production rules to the service grammar, that allow you to express more complex services (more topologies).

I assume we agree that TOSCA is denotational as well?

 

Example: Rule/trigger-based programming is compositional, because you can keep adding more rules that would âfireâ under certain conditions. But it is not denotational, because adding one rule could completely alter the meaning of existing rules. Basically, such a non-denotational rule-base gives you something akin to the proverbial Aprils fool âconditional comefromâ construct https://en.wikipedia.org/wiki/COMEFROM

 

Doesnât this example mix and match the semantics of the language vs. the semantics of the program created using the language? One can always add code to a program that changes the behavior of previous code that was already written, but hopefully that code doesnât change the semantics of what is expressed using the language?

 

-          What is it about TOSCA in your opinion that makes it not Turing-complete (as opposed to a compositional graph grammar?)

TOSCA is not trying to be a programming language (aka Turing-complete). Speculatively, I guess you might be able to make substitutions recursive, and so claim Turing completeness, but that is not a recommended use of TOSCA. Graph Grammars by themselves are not a programming language either â that would be like claiming that BNF is a programming language. So of course there is more to thatâ

 

Perhaps weâre not entirely on the same page about the objectives for TOSCA. I absolutely try to look at TOSCA as a programming language. My orchestrator doesnât know how to do anything other than what it is programmed to do by TOSCA service templates, i.e. by applying TOSCA language semantics to those templates. The TOSCA language semantics define how graph instances are created from templates (using provided input values), how requirements are fulfilled, how substitution mapping is used to decompose services, how to create workflows that ârealizeâ the resulting service topology graphs on the external platforms, how to generate events (ânotificationsâ) by monitoring external platforms, how to âtriggerâ ECA policies in response to events, etc.

Also, substitution mapping is definitely intended to be recursive. I use substitution mapping primarily to deal with abstraction: I map abstract service representations into lower-level topologies that apply configurations to external platforms. The most practical way to accomplish this is to do this recursively (by mapping âbusiness viewâ to  âsystem viewâ to âadministrator viewâ to âdevice viewâ to âinstance viewâ using John Strassnerâs policy continuum framework).

 

-          What do you mean by âcompletely declarativeâ? In my opinion, any declarative language requires a tool that turns âdeclarationsâ into the âimperativeâ sequence of steps that must be taken to ârealizeâ those declarations. If that tool has built-in knowledge about the set of entities that can be used in those declarative descriptions (e.g., Kubernetes), then nothing further is required. However, now that we have removed the Simple Profile types from the TOSCA language, TOSCA no longer has any built-in knowledge about anything, and every type is a âcustomâ type. This means that âimperativeâ support must be provided to âimplementâ those custom types. Fortunately, the TOSCA language includes imperative features that support this. The fact that TOSCA is both declarative and imperative is one of the very important differentiators of TOSCA, in my opinion:

o   Service designers use TOSCA as a declarative language, using types defined in the various TOSCA profiles

o   Profile designers use TOSCA as an imperative language by defining interfaces, operations, notifications, and artifacts that âimplementâ the types in their profiles

 

TOSCA, HOT and DSD are the same family, equally declarative, as Tal pointed out.

 

The languages are declarative in the sense that an _expression_ in that language does not say: âFirst do this, then thatâ. You can add artifacts etc, that allows an orchestrator to create a plan/playbook/process/â to deploy the service, but the orchestrator could also derive how to undeploy the same service, or modify it. So the language itself does not imply the direction of actions, only policies for creating actions after being given a direction. Compare to imperative languages like C++/Java/Workflows. There is no way to âreverseâ the actions of such languages. Functional languages are not reversible, but logic languages â Prolog â are (unless you use cuts â!â), so they can be used declaratively. The fact that there are policies/artifacts/interfaces/operations/etc that guide the translation from declarative to sets of actions does not make these languages less declarative in their essence.

 

I donât look at âundeployâ as a way to âundoâ, but rather as one possible âentry pointâ into the TOSCA service template (just like there may be multiple entry points in a Java or Python package). In fact, I maintain that there really are only 5 âcanonicalâ entry points into a TOSCA service template to which all other actions can be reduced: deploy, undeploy, update/modify, upgrade, and ârun workflowâ (there are of course also the southbound callbacks that represent notifications).

 

By the way, I am aware of the âreversible programmingâ research topic, but that is something different.

 

I would also like to come back to some of my earlier comments:

 

-          We continue to talk about âcloud-native orchestrationâ without actually ever defining that term. In order to have productive discussions, we need clear definitions about terminology.

 

Here is an initial shot at a definition:

 

The orchestration can be cloud-native if and only if:

1.       The orchestrator itself is cloud-native

While this makes a lot of sense, this feels strictly like an implementation detail, and perhaps not part of the definition for âcloud native orchestrationâ

2.       Every part of a service can be created/deleted/configured/reconfigured on a cloud-native sub-system by sending a declarative configuration to that sub-system

Does this imply then that being âdeclarativeâ is a prerequisite for being cloud-native?

3.       Any order of pushing declarations to sub-systems has the same effect (temporal symmetry)

Didnât you argue earlier that this may not always be possible?

4.       Any required coordination between the sub-systems is handled by communication directly between the sub-systems, and does not require mediation by the orchestrator

Iâm not sure I agree with this. In fact, this implies distributed coordination that may lead to exactly the types of event storms you were concerned about earlier. I believe each subsystem should only be concerned with its âlocalâ domain (which I believe is what Tal suggested). If inter-domain coordination is required, sub-domains should escalate to their âparentâ coordinator to handle inter-domain issues (since those inter-domain issues are in fact âlocalâ to the parent coordinate).

5.       Optionally (?): The orchestrator is informed by each sub-system about its state, allowing the orchestrator to represent snapshots of the end-to-end state of the service

I donât believe this should be optional 😊

6.       Optionally (?): If the orchestrator is informed about deviations from the desired end-to-end service configuration in a sub-system, it may react by pushing additional configurations to the sub-systems

Same comment here. I donât believe this is optional.

 

Tal, please improve.

 

What I may be getting wrong here is that there is still a hierarchy of âthe orchestratorâ and the âsub-systemsâ, but could there be an architecture of peer-orchestrators, where each orchestrator would appear as a sub-system to the other orchestrators? But in such an architecture, could the peer-orchestrators have conflicting âobjectivesâ so they might start âfightingâ by perceiving the configurations pushed by other orchestrators as conflicting with their own objectives?

Yes, which is why I believe coordination should be top-down, not peer-to-peer.

 

-          The same holds for the âmoving from the inside outâ (the centrifuge concept) vs. âmoving from the top downâ. I believe Tal wanted to position these as two very different orchestration concepts, but I must admit Iâm not entirely clear on the core distinction between these two. Is the âcentrifugeâ concept an implementation of the Operator pattern (where declarative descriptions are converted into more fine grain declarative descriptions) or am I missing the point? If so, I have previously suggested âdecompose and delegateâ as a way to describe the same concept. If this is what the centrifuge concept represents, then I agree that this is a âmust-haveâ feature. But I believe this feature should be used for âdecomposingâ abstract representations into concrete descriptions that can actually be orchestrated, not as the orchestration functionality itself. Either way, I believe it would be helpful to better describe these concepts.

 

This is for Tal.

 

/Peter

 

Thanks. Extremely helpful and productive!