TF-M Scheduler ============== Objectives and Requirements --------------------------- a. Decoupling of scheduling decisions between NSPE and SPE b. Efficient interrupt handling on SPE as well as NSPE c. Reduce critical sections on the secure side to not block interrupts unnecessarily d. Scalability to allow simplification/reduction of overheads to scale down the design for constrained devices e. Reduce impact on NSPE software design a. NSPE interrupt handling implementation should be independent i. Worst case latency (TBD, can be blocked by SPM that runs at highest prio) b. Impact on the NSPE scheduler should be minimal c. No assumptions should be made about NSPE's scheduling capabilities Scheduler Rules for context switching between SPE and NSPE ---------------------------------------------------------- To allow coherent cooperative scheduling, following set of rules are imposed on security state changes. The switching between SPE and NSPE can be triggered in multiple ways. **Involuntary security context switch;** when the software has no control over the switch - A NSPE interrupt take control into NSPE from SPE - A SPE interrupt takes control into NSPE from SPE **Voluntary security context switch;** when software programmatically makes the switch - A NSPE exception handler returns from NSPE to pre-empted SPE context - A SPE exception handler returns from SPE to pre-empted NSPE context - NSPE makes a function call into SPE - SPE returns a call from NSPE - SPE makes a function call into NSPE (not covered in current design) - NSPE returns a call from SPE (not covered in current design) In order to maintain the context integrity across NSPE and SPE, following rules are imposed on all security state switches. ### Rules for NSPE Exception handling a. The NSPE exception handler is allowed to trigger a NSPE context switch (regardless of security state of the pre-empted context. b. The NSPE scheduler must eventually 'restore' the pre-empted (by exception) context. This is a normal expected behaviour of any scheduler. c. If NSPE exception results in a context switch, SPM must be informed of the scheduling decision; this must be done BEFORE the execution of scheduled-in context. Possible implementation options: a. Every time NSPE scheduler changes the context it informs the SPM regardless of whether the new NSPE context would make a call to SPE or not. OR b. NSPE tags the contexts (threads) that are meant to make a call into SPE; scheduling in/out of only the tagged contexts is communicated to SPM. This approach may improve NSPE scheduler performance based on use-cases. ### Rules for SPE Exception handling a. All of the SPE interrupts have higher priority than NSPE interrupts b. The SPE interrupt handler is allowed to trigger a SPE context switch (regardless of security state of the pre-empted context c. SPE scheduler must treat pre-empted context as one of the SPE contexts- a. If the pre-empted SPE context is SP1, the TCB for SP1 should be used for saving the context b. If SP1 was pre-empted by a NSPE interrupt, and subsequent NSPE execution is pre-empted by SPE exception (\_\_before\_\_ NSPE scheduling decision is communicated back to SPM) -- SP1 TCB must be used for saving the context c. If SP1 was pre-empted by a NSPE interrupt, and subsequent NSPE execution is pre-empted by SPE exception (\_\_after\_\_ NSPE scheduling decision is communicated back to SPM) -- a TCB dedicated to NSPE should be used for saving the context d. The SPE scheduler must eventually 'restore' the pre-empted (by exception) context. This is a normal expected behaviour of any scheduler. e. All of the interrupts belonging to a partition must have same priority. i.e. ISR pre-emption within same partition is not allowed. f. In case of nested interrupts, control must NOT return to thread mode until the lowest prio ISR has finished (TBD: check -- this must be handled in the hardware) g. If the previously pre-empted context was a NSPE ISR context, SPE ISR handler MUST return to pre-empted NSPE context instead of returning to SPE thread mode. (TBD: check -- this must be handled in the hardware) ### Rules for NSPE to SPM function call (and NSPE scheduler) a. Current NSPE context must have been communicated to SPM, otherwise SPM cannot guarantee NSPE function calling stack integrity. ### Rules for Function Return from SPE to NSPE with result a. Last entry into secure world happens programmatically (**Voluntary security context switch into SPE)**, i.e. control is voluntarily given back by NSPE, either through a function call, or a context restore via 'return to SPE from NSPE'. As opposed to a SPE interrupt bringing back the execution into SPE. b. The result available on SPE side are for currently active NSPE context. c. The current NSPE call stack has not already been returned with SPM\_IDLE. ### Rules for Return from SPE to NSPE with SPM\_IDLE This is optional part of the design as it introduces significant complexity on both sides. It allows yielding of the CPU to NSPE when SPE has not CPU execution to do but it has not yet finished the previous request(s) from NSPE; i.e. SPE is waiting on arrival of a SPE interrupt. a. Last entry into secure world happens programmatically (**Voluntary security context switch into SPE)**, i.e. control is voluntarily given back by NSPE, either through a function call, or a context restore via 'return to SPE from NSPE'. As opposed to a SPE interrupt bringing back the execution into SPE. b. The result for the currently active NSPE entity is not yet available, the called service is waiting (on interrupt/event). c. The current NSPE call stack has not already been returned with SPM\_IDLE. ### Rules for NSPE pend irq based return from SPE to NSPE \[TBD\] explain pendIRQ based return This is optional part of the design as it introduces significant complexity on both sides. This works in conjunction with [Rules for Return from SPE to NSPE with SPM\_IDLE](#rules-for-return-from-spe-to-nspe-with-spm_idle). a. The SPE has finished a NSPE request. b. The corresponding NSPE context has already been returned with SPM\_IDLE. ### Rules for ISR pre-emption a. A higher priority NSPE interrupt is allowed to pre-empt a lower priority NSPE ISR b. A higher priority SPE interrupt is allowed to pre-empt a lower priority SPE ISR c. A SPE interrupt is allowed to pre-empt NSPE ISR d. A NSPE interrupt can NOT pre-empt SPE ISR e. All interrupts belonging to a service must have same priority Scheduling Scenarios -------------------- ### NSPE interrupt when SPE executing \[insert "NS interrupt"\] \- NSPE1 makes call to secure service SP1 \- NSPE interrupt NS\_ITR1 is triggered while SP1 is running \- Hardware takes control to NSPE ISR routine, secure context is pushed into SP1 stack \- NSPE scheduler schedules second NSPE context, NSPE2; a context change notification is sent to SPM before NSPE2 is run \[move to common -- context change notification handling\] \- NSPE2 makes a call to SP2 \- SPM schedules SP2 \- SP2 finishes execution, returns to SPM

TF-M Scheduler - Rules ============== On ArmV8-M CPUs, NSPE and SPE share the same physical processing element(PE). A TF-M enabled system need to be able to handle asynchronous events (interrupts) regardless of current security state of the PE, and that may lead to scheduling decisions. This introduces significant complexity into TF-M. To keep the integrity of (NSPE and SPE) schedulers and call paths between NSPE and SPE, following set of rules are imposed on the TF-M scheduler design. Objectives and Requirements --------------------------- a. Decoupling of scheduling decisions between NSPE and SPE b. Efficient interrupt handling by SPE as well as NSPE c. Reduce critical sections on the secure side to not block interrupts unnecessarily d. Scalability to allow simplification/reduction of overheads to scale down the design for constrained devices e. Reduce impact on NSPE software design a. NSPE interrupt handling implementation should be independent b. Impact on the NSPE scheduler should be minimal c. No assumptions should be made about NSPE's scheduling capabilities Scheduler Rules for context switching between SPE and NSPE ---------------------------------------------------------- To allow coherent cooperative scheduling, following set of rules are imposed on security state changes. The switching between SPE and NSPE can be triggered in multiple ways. **Involuntary security state switch;** when the software has no control over the switch - A NSPE interrupt take control into NSPE from SPE - A SPE interrupt takes control into SPE from NSPE **Voluntary security state switch;** when software programmatically makes the switch - A NSPE exception handler returns from NSPE to pre-empted SPE context - A SPE exception handler returns from SPE to pre-empted NSPE context - NSPE makes a function call into SPE - SPE returns a call from NSPE - SPE makes a function call into NSPE (not covered in current design) - NSPE returns a call from SPE (not covered in current design) In order to maintain the call stack integrity across NSPE and SPE, following rules are imposed on all security state switches. ### Rules for NSPE Exception handling **a. The NSPE exception handler is allowed to trigger a NSPE context switch (regardless of security state of the preempted context.** This is expected behaviour of any (RT)OS. **b. The NSPE scheduler must eventually 'restore' the preempted (by exception) context.** This is expected behaviour of any (RT)OS. **c. If NSPE exception results in a NSPE context switch, SPM must be informed of the scheduling decision; this must be done BEFORE the execution of newly scheduled-in context.** This is to ensures integrity of the call stack when SPE is ready to return a previous function call from NSPE. ### Rules for SPE Exception handling **a. All of the SPE interrupts must have higher priority than NSPE interrupts** This is rule is primarily for simplifying the SPM design. **b. The SPE interrupt handler is allowed to trigger a SPE context switch (regardless of security state of the pre-empted context** If the SPE context targeted by the interrupt is not same as current SPE context, the SPM may choose to switch the current running SPE context based on priority. **c. SPE scheduler must treat pre-empted context as one of the SPE contexts** **c.1 If the pre-empted SPE context is SP1, the TCB for SP1 should be used for saving the context** i.e. the context of SP1 should be saved before scheduling anything other secure partition. **c.2 If SP1 was pre-empted by a NSPE interrupt, and subsequent NSPE execution is pre-empted by SPE exception (__before__ NSPE scheduling decision is communicated back to SPM) -- SP1 TCB must be used for saving the context** In this case SPM is not yet aware of the NSPE context switch, from SPM's standpoint SP1 is still executing, so SPM assumes that the preempted context is SP1. **c.3 If SP1 was pre-empted by a NSPE interrupt, and subsequent NSPE execution is pre-empted by SPE exception ___after__ NSPE scheduling decision is communicated back to SPM) -- a TCB dedicated to NSPE should be used for saving the context** When NSPE scheduler communicates the scheduling decision to SPM, SPM must save the SP1 context, if a SPE interrupt preempts the currently running NSPE context, SPM should save the context to a dedicated NSPE TCB. **c.4 The SPE scheduler must eventually 'restore' the pre-empted context.** This is an expected behaviour of any scheduler. **d. All of the interrupts belonging to a partition must have same priority.** This serializes ISR execution targeted for same partition. **e. In case of nested interrupts, all of the ISRs must run to finish before any service code is allowed to run** This is an expected behaviour of any scheduler. **f. If the previously preempted context was a NSPE ISR context, SPE ISR handler must return to preempted NSPE context.** This is an expected behaviour of any scheduler to return to preempted context. ### Rules for NSPE to SPM function call (and NSPE scheduler) a. Current NSPE context must have been communicated to SPM, otherwise SPM cannot guarantee NSPE function calling stack integrity. ### Rules for Function Return from SPE to NSPE with result **a. The result available on SPE side are for currently active NSPE context.** To maintain call stack integrity, if SPE is ready to return to NSPE, it can do function return only if the SPE return path corresponds to currently active NSPE context. **b. Last entry into secure world happened programmatically (Voluntary security state switch into SPE)** i.e. control is voluntarily given back by NSPE, either through a function call, or a context restore via 'return to SPE from NSPE'. As opposed to a SPE interrupt bringing back the execution into SPE. **c. The current NSPE call stack has not already been returned with SPM_IDLE.** This rule applies if following optional feature is enabled. ### Rules for Return from SPE to NSPE with SPM_IDLE This is optional part of the design as it introduces significant complexity on both sides of the security boundary. It allows yielding of the CPU to NSPE when SPE has not CPU execution to do but it has not yet finished the previous request(s) from NSPE; i.e. SPE is waiting on arrival of a SPE interrupt. **a. Last entry into secure world happens programmatically (Voluntary security context switch into SPE)** i.e. control is voluntarily given back by NSPE, either through a function call, or a context restore via 'return to SPE from NSPE'. As opposed to a SPE interrupt bringing back the execution into SPE. **b. The result for the currently active NSPE entity is not yet available, the called service is waiting (on interrupt/event).** SPE request corresponding to currently active NSPE caller is not yet completed and is waiting on an ISR. **c. The current NSPE call stack has not already been returned with SPM_IDLE.** ### Rules for NSPE pend irq based return from SPE to NSPE This is optional part of the design as it introduces significant complexity on both sides. This works in conjunction with [Rules for Return from SPE to NSPE with SPM_IDLE](#rules-for-return-from-spe-to-nspe-with-spm_idle). In this scenario, when SPE is ready with result for a previous call from NSPE, it raises a pended IRQ to NSPE instead of returning the function call path. **a. The SPE has finished a NSPE request.** **b. The corresponding NSPE context has already been returned with SPM_IDLE.** ### Rules for ISR pre-emption **a. A higher priority NSPE interrupt is allowed to preempt a lower priority NSPE ISR** **b. A higher priority SPE interrupt is allowed to preempt a lower priority SPE ISR** **c. A SPE interrupt is allowed to preempt NSPE ISR** **d. A NSPE interrupt is not allowed to preempt SPE ISR** **e. All interrupts belonging to a service must have same priority**

TF-M Scheduler - Rules ============== On ArmV8-M CPUs, NSPE and SPE share the same physical processing element(PE). A TF-M enabled system need to be able to handle asynchronous events (interrupts) regardless of current security state of the PE, and that may lead to scheduling decisions. This introduces significant complexity into TF-M. To keep the integrity of (NSPE and SPE) schedulers and call paths between NSPE and SPE, following set of rules are imposed on the TF-M scheduler design. Objectives and Requirements --------------------------- a. Decoupling of scheduling decisions between NSPE and SPE b. Efficient interrupt handling onby SPE as well as NSPE c. Reduce critical sections on the secure side to not block interrupts unnecessarily d. Scalability to allow simplification/reduction of overheads to scale down the design for constrained devices e. Reduce impact on NSPE software design a. NSPE interrupt handling implementation should be independent i. Worst case latency (TBD, can be blocked by SPM that runs at highest prio) b. Impact on the NSPE scheduler should be minimal c. No assumptions should be made about NSPE's scheduling capabilities Scheduler Rules for context switching between SPE and NSPE ---------------------------------------------------------- To allow coherent cooperative scheduling, following set of rules are imposed on security state changes. The switching between SPE and NSPE can be triggered in multiple ways. **Involuntary security contextstate switch;** when the software has no control over the switch - A NSPE interrupt take control into NSPE from SPE - A SPE interrupt takes control into SPE from NSPE - A SPE interrupt takes control into NSPE from SPE **Voluntary security contextstate switch;** when software programmatically makes the switch - A NSPE exception handler returns from NSPE to pre-empted SPE context - A SPE exception handler returns from SPE to pre-empted NSPE context - NSPE makes a function call into SPE - SPE returns a call from NSPE - SPE makes a function call into NSPE (not covered in current design) - NSPE returns a call from SPE (not covered in current design) In order to maintain the contextcall stack integrity across NSPE and SPE, following rules are imposed on all security state switches. ### Rules for NSPE Exception handling a. **a. The NSPE exception handler is allowed to trigger a NSPE context switch (regardless of security state of the pre-preempted context.** b. The NSPE scheduler must eventually 'restore' the pre-empted (by exception) context. This is a normal expected behaviour of any scheduler(RT)OS. c. If NSPE exception results in a context switch, SPM must be informed of the scheduling decision;**b. this must be done BEFORE the execution of scheduled-in context. Possible implementaThe NSPE scheduler must eventually 'restore' the preempted (by exception options:) context.** a. Every time NSPE scheduler changes the context it informs the SPM regardless of whether the new NSPE context would make a call to SPE or not. ORThis is expected behaviour of any (RT)OS. b**c. NSPE tags the contexts (threads) that are meant to make a call into SPE;If NSPE exception results in a NSPE context switch, scheduling in/out of onlySPM must be informed of the tagged contexts is communicated to SPM.scheduling decision; Tthis approach may improve NSPE scheduler performance based on use-cases.must be done BEFORE the execution of newly scheduled-in context.** This is to ensures integrity of the call stack when SPE is ready to return a previous function call from NSPE. ### Rules for SPE Exception handling **a. All of the SPE interrupts must have higher priority than NSPE interrupts** b. The SPE interrupt handler is allowed to trigger a SPE context switch (regardless of security state ofThis is rule is primarily for simplifying the pre-empted contextSPM design. c**b. The SPE scheduinterrupt handler must treat pre-emptedis allowed to trigger a SPE context as ont switch (regardless of security state of the SPEpre-empted contexts-** a. IfIf the SPE context targeted by the pre-empted SPE context is SP1interrupt is not same as current SPE context, the TCB for SP1 should be used for savSPM may choose to switch the current running theSPE contextt based on priority. b**c. If SP1 was pre-empted by a NSPE interrupt, and subsequent NSPE execution is pre-empted by SPE exception (\_\_before\_\_ NSPE scheduling decision is communicated back to SPM) -- SP1 TCB must be used for saving the contextSPE scheduler must treat pre-empted context as one of the SPE contexts** c.**c.1 If SP1 wasthe pre-empted by a NSPE interrupt,SPE context is SP1, the TCB for SP1 should be used for saving the context** i.e. and subsequent NSPE execution is pre-empted by SPE exception (\_\_after\_\_ NSPE scheduling decision is communicated back to SPM) -- a TCB dedicated to NSPE should be used for saving the contextthe context of SP1 should be saved before scheduling anything other secure partition. d. The SPE scheduler must eventually 'restore' the pre-empted (by exception) context.**c.2 If SP1 was pre-empted by a NSPE interrupt, and subsequent NSPE execution is pre-empted by SPE exception (__before__ NSPE scheduling decision is communicated back to SPM) -- SP1 TCB must be used for saving the context** In this case SPM is not yet aware of the NSPE context switch, from SPM's standpoint SP1 is still executing, so SPM assumes that the preempted context is SP1. **c.3 If SP1 was pre-empted by a NSPE interrupt, and subsequent NSPE execution is pre-empted by SPE exception ___after__ NSPE scheduling decision is communicated back to SPM) -- a TCB dedicated to NSPE should be used for saving the context** When NSPE scheduler communicates the scheduling decision to SPM, SPM must save the SP1 context, if a SPE interrupt preempts the currently running NSPE context, This is a normal expected behaviour of any schedulerSPM should save the context to a dedicated NSPE TCB. e. All of the interrupts belonging to a partition must have same priority. i.e. ISR pre-emption within same partition is not allowed**c.4 The SPE scheduler must eventually 'restore' the pre-empted context.** This is an expected behaviour of any scheduler. f**d. All of the interrupts belonging to a partition must have same priority.** This serializes ISR execution targeted for same partition. **e. In case of nested interrupts, control must NOT return to thread mode until the lowest prio ISR has finished (TBD: check -- this must be handled in the hardware)all of the ISRs must run to finish before any service code is allowed to run** This is an expected behaviour of any scheduler. g**f. If the previously pre-empted context was a NSPE ISR context, SPE ISR handler MUST return to pre-empted NSPE context instead of returning to SPE thread mode. (TBD: check -- this must be handled in the hardware)must return to preempted NSPE context.** This is an expected behaviour of any scheduler to return to preempted context. ### Rules for NSPE to SPM function call (and NSPE scheduler) a. Current NSPE context must have been communicated to SPM, otherwise SPM cannot guarantee NSPE function calling stack integrity. ### Rules for Function Return from SPE to NSPE with result **a. Last entry into secure world happens programmatically (**Voluntary security context switch into SPE)**, i.e. control is voluntarily given back by NSPE, either through a function callThe result available on SPE side are for currently active NSPE context.** To maintain call stack integrity, or a context restore via 'if SPE is ready to return to SPE fromo NSPE'., As opposed to ait can do function return only if the SPE interrupt bringing back the execution into SPEreturn path corresponds to currently active NSPE context. **b. The result available on SPE side are for currently active NSPE contextLast entry into secure world happened programmatically (Voluntary security state switch into SPE)** i.e. control is voluntarily given back by NSPE, either through a function call, or a context restore via 'return to SPE from NSPE'. As opposed to a SPE interrupt bringing back the execution into SPE. **c. The current NSPE call stack has not already been returned with SPM\SPM_IDLE.** This rule applies if following optional feature is enabled. ### Rules for Return from SPE to NSPE with SPM\_IDLE This is optional part of the design as it introduces significant complexity on both sides of the security boundary. It allows yielding of the CPU to NSPE when SPE has not CPU execution to do but it has not yet finished the previous request(s) from NSPE; i.e. SPE is waiting on arrival of a SPE interrupt. **a. Last entry into secure world happens programmatically (**Voluntary security context switch into SPE)**, i.e. control is voluntarily given back by NSPE, either through a function call, or a context restore via 'return to SPE from NSPE'. As opposed to a SPE interrupt bringing back the execution into SPE. b. The result for the currently active NSPE entity is not yet available, the called service is waitAs opposed to a SPE interrupt bringing (on interrupt/event)back the execution into SPE. c**b. The current NSPE call stack has not already been returned with SPM\_IDLEresult for the currently active NSPE entity is not yet available, the called service is waiting (on interrupt/event).** SPE request corresponding to currently active NSPE caller is not yet completed and is waiting on an ISR. **c. The current NSPE call stack has not already been returned with SPM_IDLE.** ### Rules for NSPE pend irq based return from SPE to NSPE \[TBD\] explain pendIRQ based return This is optional part of the design as it introduces significant complexity on both sides. This works in conjunction with [Rules for Return from SPE to NSPE with SPM_IDLE](#rules-for-return-from-spe-to-nspe-with-spm_idle). In this scenario, when SPE is ready with result for a previous call from NSPE, This works in conjunction with [Rules for Return from SPE to NSPE with SPM\_IDLE](#rules-for-return-from-spe-to-nspe-with-spm_idle)it raises a pended IRQ to NSPE instead of returning the function call path. **a. The SPE has finished a NSPE request.** **b. The corresponding NSPE context has already been returned with SPM\SPM_IDLE.** ### Rules for ISR pre-emption **a. A higher priority NSPE interrupt is allowed to pre-empt a lower priority NSPE ISR** **b. A higher priority SPE interrupt is allowed to pre-empt a lower priority SPE ISR** **c. A SPE interrupt is allowed to pre-preempt NSPE ISR** **d. A NSPE interrupt can NOTis not allowed to pre-empt SPE ISR** **e. All interrupts belonging to a service must have same priority Scheduling Scenarios** -------------------- ### NSPE interrupt when SPE executing \[insert "NS interrupt"\] \- NSPE1 makes call to secure service SP1 \- NSPE interrupt NS\_ITR1 is triggered while SP1 is running \- Hardware takes control to NSPE ISR routine, secure context is pushed into SP1 stack \- NSPE scheduler schedules second NSPE context, NSPE2; a context change notification is sent to SPM before NSPE2 is run \[move to common -- context change notification handling\] \- NSPE2 makes a call to SP2 \- SPM schedules SP2 \- SP2 finishes execution, returns to SPM