Communication Prototype Between NSPE And SPE In Dual Core System
Updated 17 Days AgoPublic

1. Introductions

This document proposes a generic prototype of the communication between NSPE (Non-secure Processing Environment) and SPE (Secure Processing Environment) in TF-M on a dual core system.

The dual core system should satisfy the following requirements

  • NSPE and SPE are properly isolated and protected following PSA
  • An Arm M-profile core locates in SPE and acts as the Secure core
  • An Inter-Processor Communication hardware module in system for communication between NSPE core and SPE core
  • TF-M runs on the Secure core with platform specific drivers support.

1.1. Scope

This design document focuses on the dual core communication design inside TF-M.
Some changes to TF-M core/Secure Partition Manager (SPM) are listed to support the dual core communication. This document only discuss about the implementation in TF-M Inter-Process Communication (IPC) model.
The TF-M non-secure interface library is tightly coupled with mailbox and RTOS implementation. The related changes to TF-M non-secure interface library are not discussed about in detail in this document.

Some requirements to mailbox functionalities are defined in this document. The detailed mailbox design or implementations is not specified in this document. Please refer to mailbox dedicate documents.

1.2. Organization of the document

  • Chapter 2 provides an overall workflow of dual core communication between NSPE and SPE.
  • Chapter 3 lists general requirements of mailbox, from the perspective of dual core communication.
  • Chapter 4 discusses about the detailed sequence and key modules of handling PSA client call in TF-M.
  • Chapter 5 summarizes the potential changes to TF-M core/SPM to support dual core communication.

2. Overall workflow in dual core communication

An overall workflow in dual-core scenario can be described as follow

  1. Non-secure application calls TF-M non-secure interface library to request Secure service. The TF-M non-secure interface library translates the Secure service into PSA Client calls.
  2. TF-M non-secure interface library notifies TF-M of the PSA client call request, via mailbox. Proper generic mailbox APIs in HAL should be defined thus TF-M non-secure interface library can co-work with diverse platform specific Inter-Processor Communication implementations.
  3. Inter-Processor Communication interrupt handler and mailbox handling in TF-M deal with the inbound mailbox event(s) and deliver the PSA client call request to TF-M SPM.
  4. TF-M SPM processes the PSA client call request. The PSA client call is eventually handled in target Secure Partition or corresponding handler.
  5. After the PSA Client call is completed, the return value is replied to NSPE via mailbox.
  6. TF-M non-secure interface library fetches return value from mailbox.
  7. The return value is returned to non-secure application.

The interfaces between NSPE app and TF-M NSPE interface library is unchanged thus the underlying platform specific details are transparent to NSPE application.

Step 3 ~ step 5 are covered in chapter 4 in detail.

3. Requirements on mailbox communication

The communication between NSPE and SPE relies on mailbox communication implementation. The mailbox functionalities are eventually implemented by platform specific Inter-Processor Communication drivers.
This section lists some general requirements on mailbox communication between NSPE and SPE.

3.1. Data transferred between NPSE and SPE

A mailbox message should contain the information and parameters of a PSA client call. After SPE is notified by a mailbox event, SPE fetches the mailbox message from NSPE for PSA Client call processing.
The mailbox design document should define the structure of mailbox message.

The information and parameters of PSA Client call in the mailbox message include

  • PSA client call type
  • Parameters required in PSA Client call. The parameters can include the following, according to PSA client call type
    • Service ID (SID)
    • Handle
    • Input vectors and the lengths
    • Output vectors and the lengths
    • Requested version of secure service
  • NSPE Client ID. Optional. The NSPE Client ID is required when NSPE RTOS enforces non-secure task isolation.

The mailbox design document can define additional members in mailbox message to accomplish mailbox communication between NSPE and SPE.

When the PSA Client call is completed in TF-M, the return result, such as PSA_SUCCESS or the handle, should be replied from SPE to NSPE via mailbox.

3.2. Mailbox synchronization between NSPE and SPE

Synchronization and protection between NSPE and SPE accesses to shared mailbox objects and variables should be implemented.

When a core accesses shared mailbox objects or variables, proper mechanisms should protect current operations from the other core.

3.3. Support to multiple ongoing PSA client call requests in TF-M (informative)

Current TF-M implementation (commit-id 8aea0c0a) only supports single outstanding PSA client call requests from NSPE.

If the support to multiple ongoing PSA client call requests in TF-M is required in future, an optional queue can be maintained in TF-M core to store multiple mailbox objects copied from NSPE, such as mailbox messages.

4. PSA client call handling flow in TF-M

This section provides more details about the flow of PSA client call handing in TF-M.

The sequence of handling PSA Client call request in TF-M is listed as below

  1. Platform specific Inter-Processor Communication interrupt handler is triggered after the mailbox event is asserted by NSPE. The interrupt handler should assert a PendSV.
  2. In the top half of PendSV handler, the scheduler selects the next thread to run and executes normal context switch if necessary.
  3. In the bottom half of PendSV handler, mailbox handling deals with the mailbox message(s) which contain(s) the PSA client call information and parameters. Then the PSA client call request is dispatched to dedicated PSA client call handler in TF-M SPM.
  4. After the PSA client call is completed, the return value is transmitted to NSPE via mailbox.

Several key modules in the whole process are covered in detail in following sections.

  • Section 4.1 discusses about the Inter-Processor Communication interrupt handler
  • Section 4.2 introduces TF-M Remote Procedure Call module to support dual-core communication.
  • Section 4.3 specifies the mailbox tasks in PendSV handler.
  • Section 4.4 proposes the routine to reply the return value to NSPE.

4.1. Inter-Processor Communication interrupt handler

Platform specific driver should implement the Inter-Processor Communication interrupt handler to deal with the Inter-Processor Communication interrupt asserted by NSPE.
The platform specific interrupt handler should complete the interrupt operations, such as interrupt EOI or acknowledge.

The interrupt handler should call generic mailbox API(s) to check whether an inbound mailbox event has occurred. Mailbox functions should be implemented in platform vendor driver to check mailbox status in NSPE and distinguish spurious events.
The interrupt handler should assert a PendSV after the inbound mailbox event is confirmed. The triggered PendSV handler will deal with the mailbox events.

Platform specific driver should put Inter-Processor Communication interrupt into a proper exception priority, according to system and application requirements.
The proper priority setting should guarantee that

  • TF-M can respond PSA client call request in time according to system and application requirements.
  • Other exceptions, which are more latency sensitive or require higher priorities, are not blocked by Inter-Processor Communication interrupt ISR.

The exception priority setting is IMPLEMENTATION DEFINED.

It is recommended to implement mailbox time-consuming operations in PendSV handler, such as mailbox message copying from NSPE to SPE. It can decrease the response latency to other interrupts, compared with putting all the mailbox operations in interrupt handler. Refer to section 4.3 for more details of PendSV handler.

In actual mailbox implementation, Inter-Processor Communication interrupt handler can execute more mailbox processing, such as mailbox message parsing and copying, to simplify the synchronization. It may increase the time consumption in interrupt handling. Therefore, it might block the handling of other interrupts whose priorities are lower or equal to Inter-Processor Communication interrupt. As a result, it becomes more important to adjust the Interrupt-Processor Communication interrupt priority in TF-M, according to specific system and implementation requirements.

4.2. TF-M Remote Procedure Call (RPC) module

This design brings up a concept of Remote Procedure Call module in TF-M.

The RPC module sits between TF-M SPM and mailbox implementation. The purpose of RPC module is to decouple mailbox implementation and TF-M SPM and enhance the generality of entire dual-core communication.

The RPC module provides a set of APIs to TF-M SPM to handle and reply PSA client call from NSPE in dual scenario. Please refer to section 5.2.1 for API details.
It hides the details of specific mailbox implementation to TF-M SPM. It avoids modifying TF-M SPM to fit mailbox development and changes.
It can keep an unified PSA client call process in TF-M SPM in both single Armv8-M scenario and dual core scenario.

The RPC module defines a set callback functions for mailbox implementation to hook its specific mailbox operations. When TF-M SPM invokes RPC APIs to deal with NSPE PSA client call, RPC module eventually calls the callbacks to execute mailbox operations.
RPC module also defines a set of PSA client call handler APIs for mailbox implementation. RPC specific client call handlers parse the PSA client call parameters and invoke common TF-M PSA client call handlers. Please refer to section for the details of RPC PSA client call handlers.
Please refer to section 5.2.2 for the details.

4.3. PendSV handler

The mailbox handling should be added to PendSV handler in current TF-M single Armv8-M implementation in IPC model. Mailbox handling processes the inbound mailbox event(s) in the bottom half of PendSV handler. The top half of PendSV contains the original scheduling.

Mailbox handling must be executed after the original scheduling to make sure that when mailbox handling triggers the sleeping secure service, the status of sleeping secure service has been updated in scheduling.

A compile flag can be defined to disable mailbox handling in PendSV handler in single Armv8-M scenario during building.

This section only discusses about the mailbox handling in the bottom half of PendSV handler. The TF-M scheduling and context switch should keep unchanged as current single Armv8-M implementation.

4.3.1. Mailbox handling in bottom half of PendSV handler

PendSV handler should call RPC API tfm_rpc_client_call_handler() to check and handle PSA client call request from NSPE. tfm_rpc_client_call_handler() invokes request handling callback function to eventually execute specific mailbox message handling operations. The mailbox APIs are defined in mailbox design document

The handling process in mailbox operation consists of the following steps.

  1. If copy operations are not done in Inter-Processor Communication interrupt handler, the mailbox handling should copy the mailbox message(s) containing PSA client call request from NSPE. Proper protection and synchronization should be implemented in mailbox to guarantee that copy operations are not interfered by NSPE mailbox operations or Inter-Processor Communication interrupt handler. If a queue is maintained inside TF-M core to support multiple outstanding PSA client call requests, mailbox handling can copy multiple mailbox messages together into the queue, to save the time of synchronization between two cores.
  1. Mailbox handling parses the mailbox message copied in SPE and fetches the information of the PSA client call, including the PSA client call type. Additional checks can be executed to make sure that the mailbox message is valid. It may require additional information carried in the mailbox message.
  1. The PSA client call request is dispatched to the dedicated TF-M RPC PSA client call handler. The PSA client call request is processed in the corresponding handler.
    • For psa_framework_version() and psa_version(), the PSA client call can be completed in the handlers tfm_rpc_psa_framework_version() and tfm_rpc_psa_version() respectively.
    • For psa_connect(), psa_call() and psa_close(), the handlers tfm_rpc_psa_connect(), tfm_rpc_psa_call() and tfm_rpc_psa_close()create the PSA message and trigger target Secure partition respectively. The target Secure partition will be waken up to handle the PSA message.

The dual-core scenario and single Armv8-M scenario in TF-M IPC implementation should share the same PSA client call routines inside TF-M SPM. The current handler definitions can be adjusted to be more generic for dual-core scenario and single Armv8-M implementation. Please refer to chapter 5 for details.

If there are multiple NSPE PSA client call requests pending, the mailbox handling can process mailbox messages one by one. Mailbox handling can complete all the mailbox messages processing before running to the bottom half of the PendSV handler.

4.3.2. Implementation details in PendSV handler

Some more details should be taken care of in actual implementation.

  • PendSV priority should be configured as low enough, to prevent blocking or preempting other latency sensitive interrupts.
  • All the mailbox implementations inside PendSV handler must not directly execute context switch.
  • To simplify the interrupt handling inside TF-M, the mailbox handling implementation inside PendSV handle should avoid triggering additional Inter-Processor Communication interrupts in TF-M, unless it is explicitly required in mailbox design.
  • If Inter-Processor Communication interrupt handler and PendSV handler access shared mailbox objects, proper protection and synchronization should be implemented in both handlers. For example, the Inter-Processor Communication interrupt can be temporarily disabled on secure core while PendSV handler accesses mailbox objects in TF-M.

4.4. Return value replying routine in TF-M

Diverse PSA client calls can be implemented with different return value replying routines.

  • Section 4.4.1 describes the routine for psa_framework_version() and psa_version().
  • Section 4.4.2 describes the routine for psa_connect(), psa_call() and psa_close().

4.4.1. Replying routine for psa_framework_version() and psa_version()

For psa_framework_version() and psa_version(), the return value can be directly returned from the dedicated TF-M RPC PSA client call handlers. Therefore, the return value can be directly replied in mailbox handling process.

A compile flag should be defined to enable replying routine via mailbox in dual-core scenario during building.

The mailbox reply functions must not trigger context switch inside PendSV handler.

4.4.2. Replying routine for psa_connect(), psa_call() and psa_close()

For psa_connect(), psa_call() and psa_close(), the PSA client call is completed in the target Secure Partition. The target Secure Partition calls psa_reply() to reply the return value to TF-M SPM. In the SVC handler of psa_reply() in TF-M SPM, TF-M SPM should call TF-M RPC API tfm_rpc_client_call_reply() to return the value to NSPE via mailbox.
tfm_rpc_client_call_reply() invokes reply callbacks to execute specific mailbox reply operations. The mailbox reply functions must not trigger context switch inside SVC handler.

If an errors occurs in the handlers, the TF-M RPC handlers, tfm_rpc_psa_call(), tfm_rpc_psa_connect() and tfm_rpc_psa_close(), may terminate and return the error, without triggering the target Secure Partition. The mailbox implementation should return error code to NSPE.

A compile flag should be defined to enable replying routine via mailbox in dual-core scenario during building.

5. Summary of changes to TF-M core/SPM

This section discusses about the general changes related to NSPE and SPE communication to current TF-M core/SPM implementations.

The detailed mailbox implementations are not covered in this section. Please refer to mailbox related specific documents.
The platform specific implementations are neither covered in this section, including the Inter-Processor Communication interrupt or its interrupt handler.

5.1. Common PSA client call handlers

Common PSA client call handlers should be extracted out from current PSA client call handlers implementation in TF-M.
Common PSA client call handlers are shared by both TF-M RPC module in dual-core scenario and SVCall handlers in single Armv8-M scenario.

5.2. TF-M RPC module

This section describes the TF-M RPC data types and APIs.

  • Section 5.2.1 lists the data types and APIs to be invoked by TF-M SPM.
  • Section 5.2.2 lists the data types and APIs to be referred by mailbox implementation

5.2.1. TF-M RPC definitions to TF-M SPM

TFM_RPC_SUCCESS

TFM_RPC_SUCCESS is a general return value to indicate that the RPC operation succeeds.

#define TFM_RPC_SUCCESS             (0)

TFM_RPC_INVAL_PARAM

TFM_RPC_INVAL_PARAM is a return value to indicate that the input parameters are invalid.

#define TFM_RPC_INVAL_PARAM         (INT32_MIN + 1)

TFM_RPC_CONFLICT_CALLBACK

Currently one and only one mailbox implementation is supported in dual core communication. This flag indicates that callback functions from one mailbox implementation are already registered and no more implementations are accepted.

#define TFM_RPC_CONFLICT_CALLBACK   (INT32_MIN + 2)

tfm_rpc_client_call_handler()

TF-M PendSV handler calls this function to handle NSPE PSA client call request.

void tfm_rpc_client_call_handler(void);

Usage
tfm_rpc_client_call_handler() invokes callback function handle_req() to execute specific mailbox handling.
Please note that tfm_rpc_client_call_handler() doesn't return the status of underlying mailbox handling.

tfm_rpc_client_call_reply()

TF-M psa_reply() handler calls this function to reply PSA client call return result to NSPE.

void tfm_rpc_client_call_reply(void *owner, int32_t ret);

Paramters

ownerA handle to identify the owner of the PSA client call return value.
retPSA client call return result value.

Usage
tfm_rpc_client_call_reply() invokes callback function reply() to execute specific mailbox reply.
Please note that tfm_rpc_client_call_reply() doesn't return the status of underlying mailbox reply process.

5.2.2. TF-M RPC definitions for mailbox

PSA client call parameters

This data structure holds the parameters used in a PSA clietn call. The parameters are passed from non-secure core to secure core via mailbox.

typedef struct client_call_params_t {
    uint32_t        sid;
    psa_handle_t    handle;
    const psa_invec *in_vec;
    size_t          in_len;
    psa_outvec      *out_vec;
    size_t          out_len;
    uint32_t        minor_version;
} client_call_params_t;

Mailbox operations callbacks

This structures contains the callback functions for specific mailbox operations.

typedef struct tfm_rpc_ops_t {
    void (*handle_req)(void);
    void (*reply)(void *owner, int32_t ret);
} tfm_rpc_ops_t;

tfm_rpc_register_ops()

This function registers underlying mailbox operations into TF-M RPC callbacks.

int32_t tfm_rpc_register_ops(const tfm_rpc_ops_t *ops_ptr);

Paramters

ops_ptrPointer to the specific operation structure.

Return

TFM_RPC_SUCCESSOperations are successfully registered.
Other error codeFail to register operations.

Usage
Mailbox should register TF-M RPC callbacks during mailbox initialization, before enabling secure services for NSPE.

tfm_rpc_unregister_ops()

This function unregisters underlying mailbox operations from TF-M RPC callbacks.

void tfm_rpc_unregister_ops(const tfm_rpc_ops_t *ops_ptr);

Paramters

ops_ptrPointer to the specific operation structure.

tfm_rpc_psa_framework_version()

TF-M RPC handler for psa_framework_version().

int32_t tfm_rpc_psa_framework_version(onst client_call_params_t *params);

Paramters

paramsBase address of parameters.

Return

versionThe version of the PSA Framework implementation that is providing the runtime services.

Usage
tfm_rpc_psa_framework_version() invokes common psa_framework_version handler in TF-M.

tfm_rpc_psa_version()

TF-M RPC handler for psa_version().

int32_t tfm_rpc_psa_version(onst client_call_params_t *params, int32_t ns_caller);

Paramters

paramsBase address of parameters.
ns_callerA non-zero value to indcate the non-secure caller.

Return

PSA_VERSION_NONEThe RoT Service is not implemented, or the caller is not permitted to access the service.
> 0The minor version of the implemented RoT Service.

Usage
tfm_rpc_psa_version() invokes common psa_version handler in TF-M.
The parameters in params should be prepared before calling tfm_rpc_psa_version().

tfm_rpc_psa_connect()

TF-M RPC handler for psa_connect().

int32_t tfm_rpc_psa_connect(const client_call_params_t *params, int32_t ns_caller);

Paramters

paramsBase address of parameters.
ns_callerA non-zero value to indicate the non-secure caller.

Return

PSA_SUCCESSSuccess.
PSA_CONNECTION_BUSYThe SPM cannot make the connection at the moment.
Does not returnThe RoT Service ID and version are not supported, or the caller is not permitted to access the service.

Usage
tfm_rpc_psa_connect() invokes common psa_connect handler in TF-M.
The parameters in params should be prepared before calling tfm_rpc_psa_connect().

tfm_rpc_psa_call()

TF-M RPC handler for psa_call().

int32_t tfm_rpc_psa_call(const client_call_params_t *params, int32_t ns_caller);

Paramters

paramsBase address of parameters.
ns_callerA non-zero value to indicate the non-secure caller.

Return

PSA_SUCCESSSuccess.
Does not returnThe call is invalid, or invalid parameters.

Usage
tfm_rpc_psa_call() invokes common psa_call handler in TF-M.
The parameters in params should be prepared before calling tfm_rpc_psa_call().

tfm_rpc_psa_close()

TF-M RPC psa_close() handler

int32_t tfm_rpc_psa_close(const client_call_params_t *params, int32_t ns_caller);

Paramters

paramsBase address of parameters.
ns_callerA non-zero value to indicate the non-secure caller.

Return

PSA_SUCCESSSuccess.
Does not returnThe call is invalid, or invalid parameters.

Usage
tfm_rpc_psa_close() invokes common psa_close handler in TF-M.
The parameters in params should be prepared before calling tfm_rpc_psa_close().

5.3. Other modifications

The following mandatory changes are also required.

  • One or more compile flag(s) should be defined to select corresponding execution routines in dual-core scenario or single Armv8-M scenario during building.
  • PendSV priority should be configured in TF-M initialization.

Some optional changes or optimizations are listed as below.

  • The PSA client call handlers of psa_connect(), psa_call() and psa_close() can be optimized to skip asserting PendSV in dual-core scenario.

Copyright (c) 2019 Arm Limited. All Rights Reserved.

Last Author
davidhuziji
Projects
  • Restricted Project
Subscribers
None