Hi Okash,
Olivier is on holiday and once he is back next week, we can arrange something to discuss.

Thanks. It seems like we are converging. Would it make sense to set up a meeting to thrash out details? Any time this week will be preferable. Let me know what works for you. Arve and Peter are in Pacific time zone. Rest of us are based in UK I guess?

That is not what I meant. The NS SVE/FP access trap is needed to avoid saving and restoring the state when the SVE/FP registers are actively used by the secure world.

In T989#11708, @soby-mathew wrote:

Hi Arve

S-EL2 cannot lazily save and restore the non secure register state though since it cannot trap accesses by the normal world. I have not thought much about how to optimize a lazy save and restore mechanism where a lower exception level also uses lazy save and restore, but I don't think the secure world is fundamentally different from the normal world here.

S-EL2 does not need to trap access by Normal world to do a lazy save and restore. The sequence that I have in mind would be something like below:

1. EL3 switches to the S-EL2 on receipt of a FF-A call from Non Secure. SPM schedules the right S-EL1 partition with the SVE and FP trap enabled. Note that the NS FPU/SVE is still present in the registers at this point in time.
2. S-EL1 SP now tries to access FP/SVE and takes a trap to SPM in S-EL2. SPM now saves the NS FPU/SVE context and restores the S-EL1 SP FP/SVE context and disables the trap. It reenters the S-EL1 partition.
3. The partition is now able to use SVE/FP and completes its work. Returns back to SPM.
4. SPM now saves the SP FP/SVE context and restores the NS SVE/FP context. Return back to NS caller via EL3.

As can be seen, The SPM does not need to trap NS SVE/FP accesses.

That is not what I meant. The NS SVE/FP access trap is needed to avoid saving and restoring the state when the SVE/FP registers are actively used by the secure world, but not by the normal world. In the sequence you describe the lazy save and restore is only lazy when the secure world does not use the SVE/FP registers.

Hi,

on systems that don't support S-EL2, SPMC functionality mostly, if not all, resides in EL3

This is an implementation choice. E.g. OP-TEE implements an S-EL1 SPMC without needing SPMC logic at EL3 (beyond the SPMD as FF-A relayer).
If you consider the EL3 FF-A SPMC just recently added, yes most of the SPMC logic resides at EL3.

Hey Soby and Olivier, on systems that don't support S-EL2, SPMC functionality mostly, if not all, resides in EL3 right? Going by that convention, would it make sense to have SVE save and restore in EL3? We can make it part of SPMC code in EL3. For additional space we can make use of DDR carveout as Soby mentioned above. Moreover, if we make that context save and restore part enablement configurable at compile time, then platform can choose whether they want the functionality. Would it then be acceptable?

Hi,
Today SEL2 unconditionally saves/restores FP/SIMD/SVE NS context on any SEL2 entry/exit.
I believe it could be optimized the way Soby is describing it by bullets 1,2,3,4.
It is worth noting that when SEL2 is not present (e.g. using the EL3 FF-A SPMC and a SEL1 TOS), the same lazy NS and TA contexts save/restore mechanism can be used by a SEL1 TOS and EL3 doesn't have to bother.

Hi Arve

In T989#11703, @soby-mathew wrote:

When you have a S-EL2 based system with possibly multiple S-EL1 partitions, it would be complex to implement a scheme where in EL3 will restore the right S-EL1 context on taking a trap during lazy save mechanism. In such systems, it is easier for S-EL2 to implement such a scheme since it is the manager for S-EL1 contexts.

S-EL2 cannot lazily save and restore the non secure register state though since it cannot trap accesses by the normal world. I have not thought much about how to optimize a lazy save and restore mechanism where a lower exception level also uses lazy save and restore, but I don't think the secure world is fundamentally different from the normal world here. You can have lazy save and restore in NS-EL1, NS-EL2, S-EL1, S-EL2 and EL3. I think it is worthwhile to see how this can be optimized to avoid saving and restoring register states that will not be used, but I would like to see a solution that does not leak data between execution environments that are supposed to be isolated.

TF-A works on a 6 month release cadence and we typically update the gcc toolchain to the latest released version along with the TF-A release. gcc versions we use are downloaded from https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/downloads.
Currently we don't update the toolchain to any versions newer than the one released here. We used the version 10.3-2021.07 with TF-A v2.6 release and will be updating to version 11.2-2022.02 with our upcoming v2.7 release which is planned for the end of May, 2022.

When you have a S-EL2 based system with possibly multiple S-EL1 partitions, it would be complex to implement a scheme where in EL3 will restore the right S-EL1 context on taking a trap during lazy save mechanism. In such systems, it is easier for S-EL2 to implement such a scheme since it is the manager for S-EL1 contexts.

Hey Olivier,

A similar issue was reported in the ticket https://developer.trustedfirmware.org/T984
Can you confirm if this issue is due to a bug in the toolchain itself?

If you are still interested, we have an ongoing patch for adding an option to enable FPU coprocessors CP10/CP11: https://review.trustedfirmware.org/c/TF-M/trusted-firmware-m/+/15243

Hi! Thanks for your draft! I created a patch: https://review.trustedfirmware.org/c/TF-M/tf-m-tests/+/15150. Does it meet your requirements?

Reported bug in gcc 12.1.0

• https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105523

The issue is related to this GCC ticket:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99578
Which is corrected in the official GCC 12.1 version.

My modification is very draft, so I just list it here. Actually, the above has mentioned in total.

Thanks for your feedback.

Shorten the build directory -B C:/build can help, but not enough. The same error still occurs.

This issue was spotted by windows users, and here are some workarounds:

Hi Peter,
So far this has been a deliberate design choice to avoid saving/restoring SVE state from EL3 mainly for BL31 footprint reasons (and performance if unconditionally done on each and every world switch). The vector register file ranges from 2KB to 8KB with 8 cores, and linearly scales to as many cores in the system (which can be hundreds in a server chipset). Apart from specific cases under discussion (SPM-MM or EL3 FF-A SPM), it is preferable to do this at lower EL e.g. a TOS at SEL1 (or Hafnium at SEL2). Do you have specific reasons why it cannot be done at lower EL?
You may also want to take a look at those options: ENABLE_SVE_FOR_NS and ENABLE_SVE_FOR_SWD
https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git/tree/docs/getting_started/build-options.rst#n409
Regards,
Olivier.