Page MenuHomePhabricator
Create Task
Maniphest T1003

TF-A: RPi4 fails to boot secondary core at second time by PSCI cpu_on
Open, HighPublic
Actions

Description

This is my operation on RPi 4B, the firmware is RPi4_UEFI_Firmware_v1.33:
primary core boot -> PSCI cpu_on(called by primary core) -> secondary core boot(affinity_state is AFF_STATE_ON) -> PSCI cpu off (called by secondary core) -> secondary core powerdown(affinity_state is AFF_STATE_OFF) -> PSCI cpu_on(called by primary core twice) -> secondary core boot(affinity_state is AFF_STATE_ON_PENDING)

PSCI cpu_on operation:
struct arm_smccc_res res;
cpuid = 3;
secondary_entry = 0x7a000000;
arm_smccc_smc(0xC4000003, cpuid, secondary_entry, 0, 0, 0, 0, 0, &res);

PSCI cpu_off operation:
arm_smccc_smc(0x84000002, 0, 0, 0, 0, 0, 0, 0, &res);

When I boot the secondary core at second time, it returns success(res.a0 returns 0). But the affinity_state is AFF_STATE_ON_PENDING instead of AFF_STATE_ON.
In my prediction, the last affinity_state of secondary core should be AFF_STATE_ON. I tried to figure out what problem it is, do I miss some steps?

In arm-trusted-firmware\plat\rpi\common\rpi3_pm.c, I find the hook pwr_domain_pwr_down_wfi is covered by rpi3_pwr_down_wfi directly, which previously is rpi3_pwr_domain_pwr_down_wfi.
The modification is in this commit: https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git/commit/plat/rpi?h=v2.3&id=2e5f84432dabd74cbcd045e5007e02c7d3b25a91

Previously(without the commit):
pwr_domain_pwr_down_wfi->rpi3_pwr_domain_pwr_down_wfi->plat_secondary_cold_boot_setup->plat_wait_for_warm_boot
It will bring secondary core to wfe, waiting for mailbox, come into bl31_warm_entrypoint when the mailbox value is PLAT_RPI3_TM_HOLD_STATE_GO, and finally the affinity_state will be set to AFF_STATE_ON in bl31_warm_entrypoint.

Now(within the commit):
pwr_domain_pwr_down_wfi->rpi3_pwr_down_wfi->write_rmr_el3(RMR_EL3_RR_BIT | RMR_EL3_AA64_BIT)
It resets the secondary core via RMR, waiting for a "warm boot" request. But I don't know how to receive the "warm boot" request and come into bl31_warm_entrypoint. It seems it's not workable in this way. In this case, the last affinity_state is still AFF_STATE_ON_PENDING.

Event Timeline

Luoject created this task.Sep 24 2022, 11:01 AM
Luoject added a project: Trusted Firmware A.
joannafarley-arm added subscribers: Andre-ARM, joannafarley-arm.Sep 24 2022, 3:23 PM

Pinging @Andre-ARM RPi4 platform code owner for comment.

Luoject updated the task description. (Show Details)Sep 26 2022, 7:42 AM
Andre-ARM added a comment.Sep 26 2022, 9:55 AM

So that's a lot of details (thanks for that!), but what is the actual problem? That secondaries cannot be taken offline? Or that they don't came back online? And did that work before the commit you mentioned?

Luoject added a comment.Sep 26 2022, 12:09 PM
In T1003#11849, @Andre-ARM wrote:

So that's a lot of details (thanks for that!), but what is the actual problem? That secondaries cannot be taken offline? Or that they don't came back online? And did that work before the commit you mentioned?

Hi Andre-ARM, the actual problem is that secondaries is powered down successfully(the affinity state is already AFF_STATE_OFF), but after that, can not take it online again(the affinity state is AFF_STATE_ON_PENDING).

I would like to try whether it will work before the commit, but i don't know how to build TF-A, and i don't have the compiled tools to build RPI_EFI.fd. Is there any documentation to support that?

thanks!

Andre-ARM added a comment.Sep 26 2022, 1:18 PM

Hi,
thanks for the info, I will have a look at this, though probably only later this week.
As for rebuilding: I assume you are using the firmware drops from the pftf github?
This is deeply hidden in the EDK2 build system, but it will effectively use a precompiled version of bl31.bin checked into the edk2-non-osi.git repository. This will be placed into the first 128KB of RPI_EFI.fd file, filled up with 0xff.
So to replace just bl31.bin, you simply overwrite the first part of that file, with your compiled version.
To get bl31.bin from source, you just need an aarch64 (cross-)compiler, then:
$ CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi4 DEBUG=0
This should be described in docs/plat/rpi4.rst. If you find something missing, let me know, or even better: send a patch ;-)

Luoject added a comment.Sep 27 2022, 5:18 AM
In T1003#11851, @Andre-ARM wrote:

Hi,
thanks for the info, I will have a look at this, though probably only later this week.
As for rebuilding: I assume you are using the firmware drops from the pftf github?
This is deeply hidden in the EDK2 build system, but it will effectively use a precompiled version of bl31.bin checked into the edk2-non-osi.git repository. This will be placed into the first 128KB of RPI_EFI.fd file, filled up with 0xff.
So to replace just bl31.bin, you simply overwrite the first part of that file, with your compiled version.
To get bl31.bin from source, you just need an aarch64 (cross-)compiler, then:
$ CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi4 DEBUG=0
This should be described in docs/plat/rpi4.rst. If you find something missing, let me know, or even better: send a patch ;-)

Hi Andre-ARM, thanks for the building guidance.

I recovered the pwr_domain_pwr_down_wfi hook, pointed at rpi3_pwr_domain_pwr_down_wfi instead of rpi3_pwr_down_wfi, the problem was solved. In this case, RPi4 succeeds to boot secondary core at second time by PSCI cpu_on.

The problem is here: rpi3_pwr_down_wfi, which is not compatible with older function rpi3_pwr_domain_pwr_down_wfi, uses the same hook.

static const plat_psci_ops_t plat_rpi3_psci_pm_ops = {
	.cpu_standby = rpi3_cpu_standby,
	.pwr_domain_off = rpi3_pwr_domain_off,
	.pwr_domain_pwr_down_wfi = rpi3_pwr_domain_pwr_down_wfi,
	.pwr_domain_on = rpi3_pwr_domain_on,
	.pwr_domain_on_finish = rpi3_pwr_domain_on_finish,
	.pwr_domain_pwr_down_wfi = rpi3_pwr_down_wfi,
	.system_off = rpi3_system_off,
	.system_reset = rpi3_system_reset,
	.validate_power_state = rpi3_validate_power_state,
};
Luoject assigned this task to Andre-ARM.Sep 27 2022, 7:44 AM
Luoject triaged this task as High priority.
Luoject added a subscriber: andreiw.

Hi Andre-ARM, to fix the problem, here's the pull request: https://github.com/ARM-software/arm-trusted-firmware/pull/1988

Please take time to review, thanks!

• lindausa0106 added a subscriber: • lindausa0106.Jun 22 2023, 3:05 AM

I appreciate your method. I will apply it and hope that there won't be any problems. contexto

• Mike34s added a subscriber: • Mike34s.Jul 26 2023, 3:07 AM

I also ran into this error before, it stuck me for a while. Fortunately, I read your post and the word hurdle solutions you gave me helped me a lot

• herofocus added a subscriber: • herofocus.Jul 28 2023, 4:29 AM

If you only need to update bl31.bin, you can do it by replacing the first few bytes of the file with your modified binary.
You only need an aarch64 (cross-)compiler to generate bl31.bin from its source. blob opera

• niaernes added a subscriber: • niaernes.Aug 28 2023, 7:06 AM

As individuals, we are fortunate to have access to a wealth of crossover grid wisdom from various sources, ranging from experienced mentors and knowledgeable friends to online platforms and self-help literature.

• kelseyradley added a subscriber: • kelseyradley.Oct 5 2023, 9:20 AM

Replace the first few bytes of the file with your updated binary if you simply need to update bl31.bin.
Bl31.bin may be created from its source code using simply an aarch64 (cross-)compiler. Bitlife

• joniwic added a subscriber: • joniwic.Dec 19 2023, 1:16 PM

This Convertitore Mp3 is my go-to for converting audio files effortlessly. The user interface is intuitive, making the conversion process a breeze.

• Marilinas added a subscriber: • Marilinas.Dec 20 2023, 6:42 AM

Replace the first few bytes of bl31.bin with your modified binary if that's all that has to be updated. basket random

• bekean added a subscriber: • bekean.Mar 12 2024, 4:38 AM

I've shared this with my friends and infinite craft family. It's too valuable not to pass on. Let's spread the knowledge!