Re: [pkcs11] Points about AEAD and Counter mode APIs - recap from today

[Wan-Teh Chang <wtc@google.com>]

Hi Mike,

Thank you very much for your comments and proposal. I did not have
time to read your reply until the day before the F2F meeting, but I
spent an entire afternoon studying your proposal carefully and adopted
the majority of the ideas in your problem statement and proposed
solution. Unfortunately, before the F2F meeting I was only able to
prepare the slides but not a proper written response.

Last night I uploaded a document titled "PKCS #11 Message-Based
Encryption and Decryption" to the OASIS site, which should address
most of your comments. I'd like to highlight a few points in this
email.

1. My proposal allows each AEAD mechanism to specify whether any
plaintext may be released during decryption if the ciphertext or
associated data is not authentic. However, I do recommend AEAD
mechanisms not be used with multiple-part decryption functions.

2. I am afraid that you misunderstood how the message authentication
tag T in draft-mcgrew-aead-aes-cbc-hmac-sha2 can be implemented:

    T = MAC(MAC_KEY, A || S || AL),

The octet string AL (the length in bits of the associated data A,
expressed as a big-endian 64-bit unsigned integer) depends on A only.
Therefore, the crypto token can generate AL itself and does not need
the caller to provide AL as an input. So this construct does not
require us to support an associated data suffix.

Note: the A || S || AL construct is very similar to what GCM uses (see
NIST SP 800-38D, Algorithm 4, Step 5, on p. 15), so it is not a weird
construct.

I also looked into whether an associated data suffix could be useful
for authenticating network packet trailers (as opposed to headers),
but I haven't found a good use case nor an AEAD algorithm that allows
some associated data to be supplied after it has processed some
plaintext.

3. I adopted your idea of allowing existing encryption mechanisms,
such as CKM_AES_CBC, to be used with the new message-based encryption
functions. This seems like a good idea. I agree CKM_AES_CBC can be
easily made to work in message-based style, but I am worried that this
isn't the case for all existing encryption mechanisms.

4. IV generators can't be used during decryption because the
decryption side already has the IV.

5. I think I am convinced that message-based signature generation and
verification is also useful. I just thought signature generation and
verification should be easy to add once we nail down how to do
encryption and decryption. I should have noted this in my proposal.

Wan-Teh

On Sat, Feb 15, 2014 at 11:30 AM, Michael StJohns
<msj@nthpermutation.com> wrote:
> Please excuse the top posting.  I'd actually spent a bit of time commenting
> on your comments and realized that we were just moving the deck chairs
> around.  Suffice it to say that I don't think the current proposal is
> correct enough and I don't think it's easy to morph to something that is.
>
> One of the reasons for that is that I realized we actually had three issues,
> not two:
>
> The third issue I didn't call out before is the need for a per-message
> encrypt/decrypt or sign/verify.  This is different, but related to the
> per-message IV generator stuff and has a lot of the same considerations, but
> it is ultimately driven by the proliferation of network based HSMs.
> Performance for systems based on these devices is going to be constrained by
> the number of round trips necessary to do an encryption.  The classic
> Init/Update/Encrypt on a per message basis is expensive in those terms.  I
> grok this is actually more of a concern than the IV generation.
>
> So (without considering IV generation or AEAD at this point):
>
> What do we need to encrypt/decrypt each message that is common across all
> messages?
>
> a) The key
> b) The mechanism
>
> What do we need to encrypt/decrypt each message that is specific to the
> message?:
>
> a) An IV
> b) Data to be encrypted/decrypted
>
> So to make this work, we need a call to set up persistent state consisting
> of the key and the mechanism, and that state either has to be implicit to
> the session, or explicit to an object passed in to the per-message call(s).
> Wan-teh proposes this be part of the implicit session state, I'd prefer it
> to be an explicit state, but I can live with his approach.
>
> So this gets me to:
>
> C_NewAssociation (CK_SESSION_HANDLE hSession, CK_MECHANISM_PTR pMechanism,
> CK_OBJECT_HANDLE hKey);
>
> which sets up implicit session state containing the key and mechanism.  It
> also tells the session the types of calls it will be getting (so the proper
> error messages are returned if there are attempts to use intermediate
> C_EncryptUpdate calls for example).
>
> And
>
> C_EncryptMessage (CK_SESSION_HANDLE hSession, CK_VOID_PTR pIV, CK_BOOLEAN
> bLastSegment,
>     CK_BYTE_PTR pData, CK_ULONG ulDataLen, CK_BYTE_PTR pEncryptedData,
> CK_ULONG_PTR pulEncryptedDataLen);
>
> C_DecryptMessage (CK_SESSION_HANDLE hSession, CK_VOID_PTR pIV, CK_BOOLEAN
> bLastSegment,
>     CK_BYTE_PTR pEncryptedData, CK_ULONG ulEncryptedData, CK_BYTE_PTR pData,
> CK_ULONG_PTR pulDataLen);
>
> C_SignMessage (CK_SESSION_HANDLE hSession, CK_BOOLEAN pFirstSegment,
> CK_BOOLEAN bLastSegment,
>     CK_BYTE_PTR pData, CK_ULONG ulDataLen, CK_BYTE_PTR pIntegrityTag,
> CK_ULONG_PTR pulIntegrityTagLen);
>
> C_VerifyMessage(CK_SESSION_HANDLE hSession, CK_BOOLEAN pFirstSegment,
> CK_BOOLEAN bLastSegment,
>       CK_BYTE_PTR pData, CK_ULONG ulDataLen, CK_BYTE_PTR pIntegrityTag,
> CK_ULONG ulIntegrityTagLen);
>
> where
>
> pIV is the pointer to the iv, format of this defined by the mechanism (doing
> it this way to allow for IV generator later).  If iv is non-null, then the
> current segment is the first segment of the message
>
> bLastSegment is a boolean which indicates this is the last segment of
> message and that padding etc should be done prior to emitting the last of
> the cipher text.
>
> if pIV is null (or pFirstSegment is false) and bLastSegment is false, this
> is equivalent to an update operation.
>
> if pIV is non-null (or pFirstSegment is true)  and bLastSegment is true,
> this segment is the complete message.
>
> All of the above act pretty much as macros for a C_*Init/C_*Update/C_*Final
> set of calls.
>
> C_NewAssociation uses the same calling conventions as C_EncryptInit.
>
> C_CloseAssociation (CK_SESSION_HANDLE hSession) is used to close and discard
> persistent key and mechanism state.
>
>
> Second issue - AEAD ciphers.   Two sub cases - generic (Mods to
> C_EncryptInit/C_EncryptUpdate/C_EncryptFinal calls) and message (Mods to
> C_EncryptMessage).
>
> The first case is one not addressed by Wan Teh - how to fit AEAD ciphers
> into the current non-association model.
>
> AEAD is complex because of the wide range of possible approaches (sign over
> ciphertext - i.e. after encryption, sign over plain text, AAD before
> encrypted data, AAD after encrypted data, AAD mixed in with encrypted data)
> and side issues like McGrew's API proposals.    Also, there are a pair of
> functions that already exist that are somewhat applicable
> C_SignEncryptUpdate and C_DecryptVerifyUpdate.  For below I've left sign
> before encrypt vs encrypt before sign as part of the mechanism definition -
> so there's no indication of the order in the following calls.
>
> Again leaving the IV generation issues to later and dealing first with the
> current API model we create two new initialization functions and two new
> final functions:
>
> C_SignEncryptInit (CK_SESSION_HANDLE hSession,  CK_MECHANISM_PTR pMechanism,
> CK_OBJECT_HANDLE hKey,
>         CK_VOID_PTR pIV,
>         CK_BYTE_PTR aadPrefixData, CK_ULONG aadPrefixDataLen);
>
> C_DecryptVerifyInit (CK_SESSION_HANDLE hSession,  CK_MECHANISM_PTR
> pMechanism, CK_OBJECT_HANDLE hKey,
>         CK_VOID_PTR pIV,
>         CK_BYTE_PTR aadPrefixData, CK_ULONG aadPrefixDataLen);
>
> In the above, I've called out the IV as an explicit argument.  Doing it the
> way I did above means that we can probably define mechanisms that work both
> with an IV generator and with a supplied IV.
>
> The existing C_SignEncryptUpdate and C_DecryptVerifyUpdate can be used to
> process intermediate data.
>
> The final calls are used to pad, encrypt, output and to emit the integrity
> tag (or the reverse order for decrypt/verify).
>
> C_SignEncryptFinal (CK_SESSION_HANDLE hSession,
>         CK_BYTE_PTR aadSuffixData, CK_ULONG aadSuffixDataLen,
>         CK_BYTE_PTR pLastEncryptedPart, CK_ULONG_PTR
> pulLastEncryptedPartLen,
>         CK_BYTE_PTR pIntegrityTag, CK_ULONG_PTR pulIntegrityTagLen);
>
> C_DecryptVerifyFinal (CK_SESSION_HANDLE hSession,
>         CK_BYTE_PTR aadSuffixData, CK_ULONG aadSuffixDataLen,
>         CK_BYTE_PTR pLastDecryptedPart, CK_ULONG_PTR
> pulLastDecryptedPartLen,
>         CK_BYTE_PTR pIntegrityTag, CK_ULONG ulIntegrityTagLen);
>
>
> The AAD can preceed and/or follow the data to be encrypted.  That means that
> a generic API can handle McGrew's weird  AAD | ciphertext | L(AAD) construct
> as an extension to the generic CCM and GCM models.  The mechanism MAY
> constrain this, but I wouldn't define any that does.
>
> Continuing on with AEAD,  for the message models we get  new message
> functions (same C_NewAssociation, C_CloseAssociation functions plus):
>
>
> C_EncryptSignMessage (CK_SESSION_HANDLE hSession, CK_VOID_PTR iv, CK_BOOLEAN
> bLastSegment,
>     CK_BYTE_PTR pPrefixAADData, CK_ULONG ulPrefixAADDataLen,
>     CK_BYTE_PTR pData, CK_ULONG ulDataLen, CK_BYTE_PTR pEncryptedData,
> CK_ULONG_PTR pulEncryptedDataLen
>     CK_BYTE_PTR pSuffixAADData, CK_ULONG ulSuffixAADDataLen,
>     CK_BYTE_PTR pIntegrityTag, CK_ULONG_PTR pulIntegrityTagLen);
>
> C_DecryptVerifyMessage (CK_SESSION_HANDLE hSession, CK_VOID_PTR iv,
> CK_BOOLEAN bLastSegment,
>     CK_BYTE_PTR pPrefixAADData, CK_ULONG ulPrefixAADDataLen,
>     CK_BYTE_PTR pEncryptedData, CK_ULONG ulEncryptedData, CK_BYTE_PTR pData,
> CK_ULONG_PTR pulDataLen
>     CK_BYTE_PTR pSuffixAADData, CK_ULONG ulSuffixAADDataLen,
>     CK_BYTE_PTR pIntegrityTag, CK_ULONG_PTR pulIntegrityTagLen);
>
> where
>
> pIV is the pointer to the iv, format of this defined by the mechanism (doing
> it this way to allow for IV generator later).  If iv is non-null, then the
> current segment is the first segment of the message
>
> bLastSegment is a boolean which indicates this is the last segment of
> message and that padding etc should be done prior to emitting the last of
> the cipher text.
>
> if pIV is null and bLastSegment is false, this is equivalent to an update
> operation.
>
> if pIV is non-null and bLastSegment is true, this segment is the complete
> message.
>
> AAD can be both prefixed and suffixed to the encrypted message.
>
>
>
> Finally, we get to actually the least interesting of all of these - IV
> generation.
>
>
> One of the things I'm concerned with is that the IV generation model differs
> from protocol to protocol, rather than from mode to mode.  So defining one
> IV generator as part of the cryptographic mode model seems to short sighted.
> And it also doesn't work all that well if you consider that the decrypt
> operation can't always generate the IV, but is getting it as supplied in the
> message (or derived from protocol elements from all over the message).
>
> This is where my specification of "CK_VOID_PTR pIV" above comes in:
>
> 1) Existing mechanisms that specify the IV as part of the mechanism,
> continue to take their IVs from the mechanism structure for the non-message
> functions.  For the existing mechanisms using the message functions, the
> length of the IV is derived from the mechanism  or parameter data and the
> per-message IV is specified as if bExplicitIV = CK_TRUE were specified (see
> 2 below).
>
> 2) New mechanisms that want to use an IV generator will include these two
> fields in their mechanism params:
>
> a) CK_BOOLEAN bExplicitIV
> b) CK_ULONG ivLength
>
> If bExplicitIV is false, then pIV on the init call is a pointer to handle
> for an IV generator object.
>
> if bExplicitIV is true, then pIV on the init call is a pointer to a CK_BYTE
> - an array of IV bytes of length ivLength.
>
>
> IV generator objects are session storage objects.  They have a specific
> generator mechanism and specific set of attributes that are set at creation
> (and can be reset for new associations).  They have a way to read the next
> generated and current IVs (for inclusion in the protocol message for
> example).
>
> New types of generator objects can be defined as new protocols are written -
> but for the start we will define the pseudo-standard CCM (Appendix A counter
> format function), GCM and CTR generic models.
>
>
> This approach
>
> a) allows for any length of message or encrypted stream (e.g. while the
> mechanism may restrict this, the API doesn't)
> b) makes no assumption about whether or not decrypted data may be released
> before the integrity tag is verified. (again, mechanism or mode of HSM may
> restrict this).
> c) has no requirement for pre-call marshalling and formatting of the crypto
> stream (e.g. no assumptions about exactly where the integrity tag is and its
> length nor where the AAD is and its length - no memory to memory copies
> should be required prior to the call).
> d) can deal with AAD preceeding and following the ciphertext/data to be
> encrypted (future proofing)
> e) handles message based encryption as a separate and backwards compatible
> item from AAD and IV generation. (e.g. can actually do per-message CBC stuff
> within an association using new calls).
> f) divorces IV generation specification  from mechanism specification - same
> mechanism can use multiple IV generation mechanisms (future proofing).
> g) mirrors, extends or combines existing functions so easy transitions for
> future programming.
> h) possible to use IV generator for decrypt, but not required.
> i) includes functions to do message verification and signature
>
>
> My general desire here is that we don't want to have to re-do these things
> because someone has come up with yet another weird AEAD approach and our
> APIs won't support it.  I want to be able to break things down to composable
> building blocks.   I also want to stay as close to the original spec (e.g.
> for CCM and GCM the NIST documents) as possible rather than possibly niche
> documents like the McGrew API.
>
> Or to put it another way - the difference between an engineer and a
> bureaucrat is that an engineer takes large insolvable problems and breaks
> them down into small solvable pieces, where a bureaucrat takes a bunch of
> small solvable pieces and combines them into one large insolvable mass.
> I'm trying to keep the pieces useful.
>
> Mike