This is the running minutes and actions for TF-A LTS

Date: tbd
Attendees:
Minutes:

• List Item

Date: Jan 17, 2023
Attendees: Okash Khawaja (Google), Varun Wadekar (NVIDIA), Bipin Ravi (arm)

Minutes:

1. File a TFC ticket for LTS regression support planning and support with Paul's help. (Bipin)
   1. Request for support of sending emails to maintainers based on monitoring script encountering a positive score. Raised in the existing TFC ticket.
2. Invite Paul to the LTS meeting on Jan 24 or a future meeting based on availability to discuss these requirements in more details and discuss options.
3. LTS incoming patch review and populate to spreadsheet (Varun)
   1. Preliminary review with TF-A TL and platform owners ongoing
   2. The google spreadsheet been ported to https://developer.trustedfirmware.org/w/tf_a/tf-a_lts_meeting_minutes/tracking/
4. Start cherry picking applicable patches to LTS branch (Varun)
   1. Phase 1 Errata and Security patches
   2. Phase 2 Highlighted ones needing clarification with TFA TL or platform owners
   3. Platform owners to step up and voice their requirements before LTS code freeze.
   4. All cherry-picked patches to have comments field indicate why this was picked for LTS
5. Tech Forum slide deck template (Okash)
   1. Shared with Varun and Bipin
   2. Enter all applicable thoughts on the shared presentation and review by Jan 25.
   3. One suggestion for patches to be considered for LTS is using the LTS tag in Change ID subject line

Date: Dec 06, 2022
Attendees: Okash Khawaja (Google), Varun Wadekar (NVIDIA), Bipin Ravi (arm)

Minutes:

1. Solved most issues with the monitor script with help from Paul.
   1. Patch arrival notification, invoking script is OK
   2. Sending email part still not taken care and to be checked with Paul.
2. Sandbox creation for running CI/CD still NOT clear.
3. Okash agreed to make sure all pieces exist and work together
4. Varun still doesn't have access to the ticket with Paul. Issue not with NVIDIA network or the VPN.
5. Paul's patches
   1. https://review.trustedfirmware.org/c/next/ci/tf-a-job-configs/+/18013
   2. https://review.trustedfirmware.org/c/next/ci/tf-a-job-configs/+/18012
   3. https://review.trustedfirmware.org/c/next/ci/tf-a-job-configs/+/18009
   4. https://review.trustedfirmware.org/c/next/ci/tf-a-job-configs/+/17860
6. LTS branch creation
   1. Create branch for TF-A, TFTF and OpenCI scripts based of v2.8 - Bipin to create.
   2. Send email about LTS release candidate branch to community - Matteo to be part of the communication to the community
   3. Still targeting first week of Feb for the LTS release.
   4. Linaro CI bandwidth concerns to be taken with Matteo/Joanna
   5. CI run using TFTF LTS version(lts-v2.8) branched off the v2.8 release.
7. Multiple fixes coming in after v2.8.
   1. Agreed we will stick with errata and security fixes only
   2. Communicate any other fixes that addresses main functionality issues on a case-by-case basis if it happens. But "fix" doesn't mean bug fixes most of the time. - Bipin
8. Central document with 3 columns to track patches taken/rejected from LTS branch despite a positive monitor script score.
   1. To have Patch, status (accept or reject) and comments for rejection at a minimum
   2. Maintainers will have full access, other have read/comment access.

Date: Oct 25, 2022
Attendees: Okash Khawaja (Google), Varun Wadekar (NVIDIA), Bipin Ravi (arm), Joanna Farley (arm), Matteo Carlini (arm)

Minutes:

1. Define what we want to do with the automation.
   1. Automated script to identify the patches that qualify for LTS branch
      1. How to identify a patch being a CPU errata implementation from the commit header/message and patch stack? – (Bipin)
      2. Start with CPU Errata identification script design. Initially list all the steps. Target to have an early implemented version for this script by TF-A v2.8 (Okash).
      3. Add additional capabilities like Security, Platform related errata/bug fixes patches identification to be added after the CPU errata. TBD
   2. Automated script for integration to LTS branch
      1. What’s the language currently used to add hooks? – (Bipin)
      2. Git API to find the patch series? – TBD
      3. TF.org, auto merger logic etc. – Need help
2. Overall Flow
   1. Whatever we change is posted in TF-A integration branch.
      1. To track commit header, commit message and files changed. Identify keywords to generate alerts on a match. Check “CPU Errata patch identification” section below for details.
         1. Assumption is patch owners/reviewers make sure to enforce/gate keep using the right tags in commit messages/header at least moving forward.
         2. For missed patches, we need to identify the right keywords/mechanism to identify them and enhance the script as we go on.
      2. Generate an early email alert to LTS maintainers
      3. All merge conflicts are resolved in TF-A integration. So, LTS to use TF-A master branch as the base to pick final matched patches.
   2. Cherry pick errata to LTS branch from TF-A master branch once the patch is merged.
      1. Auto merge if possible, to the LTS integration branch. How do we do it? TBD
      2. Auto run the CI, plan nightly Open-CI runs.
      3. Automated email alerts for the CI results to the LTS maintainers.
3. LTS merge process
   1. Keep it simple.
   2. 2 weeks window from LTS candidate to LTS tag
   3. Start off by having 3 emails
      1. First email when creating the candidate. Interested partners can run their platform regressions with the LTS RC.
      2. Second email middle of the 2-week period warning about the approaching LTS tag(major/minor)
      3. Third email announcing the LTS tag
4. Additional relevant patches/errata to be discussed for identification
   1. Security
   2. Platform bug fixes
   3. GIC errata
5. Backporting concerns
   1. Porting patches identified for LTS branch and an effort to backport to a very old LTS tag, theoretically increasing complexity to get it merged or even impossible.
   2. Need to discuss this more but lower chances initially to encounter this.
6. Script ownership/maintainership to be shared by Arm, Google and NVIDIA.
   1. Currently no Linaro involvement
7. Can we assume all CI scripts and other infra can be used from Linaro?
   1. For TF.org, Jenkins, Infra support etc. file ticket and can request Linaro to help.
8. Current LTS assigned maintainers do not have the expertise in gerrit administrative efforts including configuring.
   1. Need to discuss this further and find a solution.
9. CPU Errata patch identification
   1. Specific changes in the patch diff to be looked for
      1. assembly macros within the <cpu>.S file within the patch is also checked in diff
      2. New line with first word “report_errata” followed by second word “ERRATA_(wildcard)” string in lib/cpus/aarch../*.S file associated with the patch.
      3. New line within the docs/design/cpu-specific-build-macros.rst having the string “`ERRATA_(wildcard)`”
      4. lib/cpus/cpu-ops.mk part of the patch hold the same “ERRATA_(wildcard)” string
      5. For CPUs, the errata enable compile flag follows the format ERRATA_<CPUNAME>_<ERRATA_ID> which could be useful later.

Date: Oct 18, 2022
Attendees: Okash Khawaja (Google), Varun Wadekar (NVIDIA), Bipin Ravi (arm), Joanna Farley (arm), Doug Richmond (arm)

Minutes:

1. LTS Branch will be created after the V2.8 release at the end of November.
2. Must also branch TFTF and the CI.
3. Target for the formal release version of the LTS branch will be at the beginning of February actual date TBD.
4. Partner testing will be performed on the LTS candidate on their downstream platforms during December and January to ensure that the LTS branch is considered stable. This is supposed to cover the extended regressions on platforms outside of TF-A. During this time the LTS should be labeled as an RC version until conclusion of testing and agreement that the formal first LTS release can be approved and published.
5. Only Bugs and fixes approved by the maintainers should be approved to be included in the LTS up to the point of its first version.
6. The versioning scheme needs to be defined. The first release in February could be versioned as lts-2.8.1 but needs agreement that this versioning does not have the potential of causing confusion with any consumers.
7. Any fixes to the LTS will only be fixes to the published functionality within either common or platform code. If there are corner cases / exceptions to this policy, then it will be the responsibility of the maintainers to decide if the code is merged from TF-A main to the LTS branch.
8. All changes required by the LTS will be tested and merged to TF-A main before being considered for inclusion in the LTS. Any exception to this requirement needs to be further thought about and discussed.
9. A mailing list needs to be created for the LTS branch with all LTS interested partners and maintainers to ensure good communication for the branch status.
10. Errata and other fixes required by the LTS from TF-A main will need to be identified, tested, approved by maintainers, and merged into the LTS. This process needs to be made as efficient as possible to reduce the effort required.
    1. To allow this process to be made efficient scripts will need to be developed to highlight changes (security fixes, bugs or errata) to cherry pick the changes from TF-A Main and trigger CI tests (post commit hook to auto merge into another branch and kick off runs) and automatically email the maintainers for their manual approval prior to automatic merging to the LTS.
    2. The manual approval is to maintain the quality of the LTS in case of patches not being correctly identified by entry of keywords for inclusion in the LTS branch.
    3. This design requires more discussion, and a follow up design is to be created with detail for the scripts which will need development prior to the LTS being published.
    4. Resource for the script development needs to be named. There is currently no development resource to perform this work.
    5. Where does the script run?
11. Errata for all CPUs needs to be included in the LTS branch. If a new partner was to start using the LTS for resident LTS CPUs, it could cause significant effort to add all the applicable errata if they were not present in the first place. Only adding critical errata for CPUs being used for current interested partners using the LTS on platforms could cause other issues with the branch configuration quality.
12. The LTS life span is understood to be 5 years (with interest to extend up to 7 years which needs further thoughts) and all LTS participating partners and arm are committed to provide maintainer support for this period. Arm has stated their named resource for Year 1 and subsequent years are to be confirmed after this pilot period.