I tried on my side with steps above, and I still cannot reproduce your issue.
Please confirm NSACR register is configured correctly, and check the FP related registers before the FP instruction causing crash.

Hi Alamy,

Hi Sherry,

I tried this on my side on STM32L562e_dk board with TF-M repo: enable "-mfloat-abi=softfp" in NS side and use TF-M default "-mfloat-abi=soft" option, and changed FP registers in non-secure thread first, then do PSA call. When system is in secure thread mode, no crash found even FPCCR_S.LSPACT = 1.

Do you have FP instruction in secure thread? If there is no FP instruction, the processor will not stack FP registers even LSPACT flag is set.
I tried this on my side on STM32L562e_dk board with TF-M repo: enable "-mfloat-abi=softfp" in NS side and use TF-M default "-mfloat-abi=soft" option, and changed FP registers in non-secure thread first, then do PSA call. When system is in secure thread mode, no crash found even FPCCR_S.LSPACT = 1.

TF-M currently support "-mfloat-abi=soft" as default, but doesn't support the setting in Zephyr "-mfloat-abi=softfp". Because they are totally different things.

But why? These ABIs are link-compatible, I do not see the problem. GCC allows these binaries to link successfully.

I mean TF-M itself current FP option is "-mfloat-abi=soft", not the "-mfloat-abi=softfp" which is used in Zephyr. It doesn't matter with setting for Zephyr.

TF-M currently support "-mfloat-abi=soft" as default, but doesn't support the setting in Zephyr "-mfloat-abi=softfp". Because they are totally different things.

Thanks, I submitted the following based on CJKay's patch above:
https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/9990

This is the explanation for "FP_SOFTABI" in Zephyr.
"config FP_SOFTABI
bool "Floating point Soft ABI"
help

	  This option selects the Floating point ABI in which hardware floating
	  point instructions are generated but soft-float calling conventions."

Yes, it crashes even without Lazy Stacking. It is a bit more deterministic as is. It crashes in the first secure exception entry, after the transition to secure domain from non-secure.

But for Zephyr, are you using soft FP or hardware FP?

The first thing I want to confirm is "I compile zephyr and Tf-M with soft FP". As you know, TF-M is default with soft FP.

1. But for Zephyr, are you using soft FP or hardware FP?
2. Is it possible for you to share the compile options and linker options for the source file including the "Non-Secure interrupt" crashing?
3. Is it possible to show the assembly code for the "Non-Secure interrupt"?
4. Please have a try to disable lazy FP stacking, to see whether still crash.

Hi Feder,

The problem has been solved.

If you didn't change TF-M while integrating into your project, PSA call(handler mode) cannot be interrupted by non-secure interrupt like you mentioned, the reason is non-secure exceptions are de-prioritized (AIRCR.PRIS = 1) in TF-M.
Non-secure interrupt can only be active when system in thread mode.

Hi Alamy,

Merged.

Hi @ioannisg,
FYI. Feder is on holiday and will back to office on 10th May.
Thanks.

Yes, the __sramdata in the declaration is a mistake, the correct target section for that global needs to be .pmusram.data. This used to be in .sram.data once upon a time but then the suspend.c stuff got added and required it to be moved to PMUSRAM. I guess they forgot to update that part in the declaration and since the old GCC seemed to silently prefer the attribute in the definition, nobody noticed. Please apply CJKay's first patch to fix the warning.

I can only assume GCC <11 simply didn't have this warning, but it looks legitimate.

Ok thanks.

It's native aarch64, the command line I'm using is:
make HOSTCC="gcc $RPM_OPT_FLAGS" CROSS_COMPILE="" PLAT=rk3399 bl31

Is this building natively on arm64 host?

I'm using Fedora 34/35 with the distribution toolchain (gcc 11.1 GA)

Can you please provide the link to the gcc11 toolchain you use?
AFAIK latest public Arm release of the cross compiler toolchains is 10.2 (https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads)

Setting this to High for now - but feel free to re-triage this was not appropriate.

Hi Feder,

For the concern you mentioned, yes, we need to add extra steps in secure scheduler, I am still working on this part.

1. When secure doing secure calls: a. if lazy fp is disabled, hardware will push/pop FP context automatically during exception entry/return. b. if lazy fp is enable, for isolation 1, secure scheduler will save and restore FP context, but not invalidate FP context; for isolation 2 and 3, secure scheduler will trigger lazy fp stacking, hardware will push FP context to thread' stack and invalidate them automatically.
2. When non-secure doing secure calls, non-secure side will SG to secure world in tfm_nspm_thread_entry, then doing secure calls as same as above. FP context of non-secure can be restored when bxns lr to non-secure side.

At a quick glance this seems to be an m2r bug: https://github.com/sphinx-doc/sphinx/issues/8705. m2r seems to be abandoned, switching to m2r2 might be a workaround, as that seems to have a fix merged.
For now the best might be to stick to the documented package versions. I am wondering if the requirements file should be more strict on Spninx version.

Hi Minos,
Is this a compatibility issue and the TFM doc build system can be upgraded, or this a bug in the latest version of Sphinx?

Hi Minos,

Hi Martina.

The first patch for this:
https://review.trustedfirmware.org/c/TF-M/trusted-firmware-m/+/9730

Currently GNU 10-2020-q4-major cannot support CMSE well (https://git.trustedfirmware.org/TF-M/trusted-firmware-m.git/tree/docs/getting_started/tfm_sw_requirement.rst#n54).
So this GNU version won't be supported in TF-M.

I built with

In T911#10554, @davidhuziji wrote:

I will run the GCC version later to check if there are more warnings. Will collect all the warnings and fix them in one shot later.

Would fix after the release. Mark as a long term goal.

Hi Martin. I was referring to PSA FF-M, not FF-A. See the spec here, in Section 2.1. TF-A conforms to FF-A, which is a different spec.

Thanks @adrianlshaw for your comment. A multi-core TF-M design is indeed a different thing than this twin-cpu design or any of the implemented designs for multiple CPUs in TF-M.
Do I understand you correctly, that such a multi-core TF-M design would not be possible to design according to PSA FF-A guidelines, because no mutex is defined in these?
How is this problem handled in TF-A as the same PSA guidelines apply there?
I somehow messed up the two files and got confused

Okay, I run into a fault when apply this patch on AN521, may need some investigations :
Running Test Suite Core non-secure interactive tests (TFM_CORE_TEST_2XXX)...

Executing 'TFM_CORE_TEST_2001'

Description: 'Interactive tests'

Segmentation fault

You will need to apply this patch in order for the interactive tests to function as without it the secure part will never build with that part enabled https://review.trustedfirmware.org/c/TF-M/tf-m-tests/+/9431
After applying that and building using:

cmake -DTFM_PLATFORM=lairdconnectivity/bl5340_dvk_cpuapp -GNinja -DTFM_TOOLCHAIN_FILE=../toolchain_GNUARM.cmake -DTEST_S=off -DTEST_NS=on -DCMAKE_BUILD_TYPE=debug -DTFM_INTERACTIVE_TEST=on -DTFM_PERIPH_ACCESS_TEST=ON -DTFM_IRQ_TEST=ON ..

And running it, the first test passes but the freezes at the same point as yours does, it does not continue with the next task

I took a look at this test case, the interactive test suit is disabled by default. Could you append some logs or command on your platform of TFM_CORE_TEST_2001
The test from my side on AN521 will stop after scenario 1:

Executing 'TFM_CORE_TEST_2001'

Description: 'Interactive tests'

Scenario 1 - SequentialTrying to acquire the TFM core from NS [seq_task]NS Lock: acquired [seq_task]Secure call to tfm_spm_core_test_sfn_veneer(&in_vec, 1, NULL, 0) failed, generic!NS Lock: releasing... [seq_task]Scenario 1 - test finished

If the NSPE can use all the ARMv8-M processors, then it makes sense to use a model similar to ARMv8-A. In that model, the request for a secure service is usually handled by the local core - no cross core interaction.

Imo, the implementation can be very platform specific. It is required that the two cores shall be physically isolated.
Besides, when NS requires secure services on the other core, the other core must runs in S world. It is a bit difficult to guarantee.

The concept I am working on would allow for execution of two NS environments both with access to TF-M services. The goal is also to isolate the NS environments from one another to provide safety in case of failure / maliciousness of one NS env. But this safety feature is not necessary for the use case of two Cortex M-33 with NS + TF-M.

@oyvind.ronningstad@nordicsemi.no This issue is on the Nordic platform. Can you take a look at it?

Assigned to Ken for the warning fix.

@MartinSchoenstedt so are you trying to run Non-secure OS + TF-M on both MPS2 AN521 cores?
Are you designing a SMP (symmetric multiprocessing) system for both NS and S? May I know the benefit to run NS + TF-M on both cores, compared to running a single Cortex-M33?

I solved this issue by myself.
The assert state works normally.

Yes this was indeed what I was thinking about. I am now trying to modify the secure enclave implementation to work with both CPUs in the SSE 200 on the MPS2 AN521 image. This would also also both cores to still be used by nonsecure OS / applications

Hi David. What about a scenario where non-secure applications want to use both cores? I think this is what Martin is asking about. Dedicating one M33 to act as a secure enclave can be considered a waste of compute resource (it will be idle most of the time).