2024.05.29

[seanmcilroy29]

---

2024.05.29 Agenda/Minutes

---

Time: Bi-weekly @ 1700 (BST) - See the time in your timezone

• Co-Chair - Chris Xie- (Futurewei)
• Co-Chair - Vacant
• Convener – Sean Mcilroy (Linux Foundation)

Antitrust Policy

Joint Development Foundation meetings may involve participation by industry competitors, and the Joint Development Foundation intends to conduct all of its activities in accordance with applicable antitrust and competition laws. It is, therefore, extremely important that attendees adhere to meeting agendas and be aware of and not participate in any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws.

If you have questions about these matters, please contact your company counsel or counsel to the Joint Development Foundation, DLA Piper.

Recordings

WG agreed to record all Meetings. This meeting recording will be available until the next scheduled meeting.
Meeting recording link

Roll Call

Please add 'Attended' to this issue during the meeting to denote attendance.

Any untracked attendees will be added by the GSF team below:

• Full Name, Affiliation, (optional) GitHub username

Agenda

• Approve agenda
• Approve previous Meeting Minutes

Introductions

• Any new members?

Discussion

• Geekbench crowdsources CPU performance matrix. Should SCER explore similar crowdsourcing approach to LLMs carbon footprint/efficiencies? #56
• Create a demo website/github pages to demonstrate the use of SCER for LLMs specification and the SCER Certification Program #55
• Create a SCER Certification/badge Program as noted in the Readme file #54

Project Dashboard

• Follow up discussion on partner outreach for LLM model benchmarking and methodology. #42
• Follow up from prior meeting minutes: further discussion on this "one potential opportunity" #41
• Add a meeting agenda: presenting SCER for LLMs in AI_Dev conference Responsible AI track for Sustainable AI life cycle #34
• Establish an instant-messaging group for real time brainstorming for research #33
• Hugging Face uses CodeCarbon to benchmark LLMs energy consumption in kWh #32
• Discuss benchmarking for LLMs #39
• Research for current ways of measuring CO2 emissions of LLM's #25
• SCI score for SCER #23
• Benchmarking #12

AOB

• Standards WG have requested for a deep dive on this at their next meeting - 06th June
• Request to present this to the SC at the next meeting - 20th June

Future meeting Agenda submissions

• If you have any Agenda submissions, please let Sean know via email [email protected] or open an Issue

Previous meeting AI's

Action Items

• Jae-Won to finish updating the model energy benchmark data from AWS with new measurements
• Jae-Won to provide thoughts on software categorization, benchmark definition and reading definition for Sheffau
• Chris and Sean to explore applying ratings to sample models from Zeus' dataset
• Jae-Won, Sean and Chris to continue discussions on the next steps for collaboration with the University of Michigan

Next Meeting

• 12th Jun @ 9 am P/T

Adjourn

• Motion to adjourn

---

Meeting Action Items / Standing Agenda / Future Agenda submissions

• Add here

[sk16-dev]

Attended

[chrisxie-fw]

attended

[seanmcilroy29]

Attended

[adityamanglik]

attended

[smazziotta]

attended

[jawache]

Attended

[sk16-dev]

Hi all, made these quickly as replacements for A. B, and C. They would go Gold, Silver, and Certified based on percentile scores.

Let me know if use of the logo for a draft is not advisable and we can change it.

@chrisxie-fw @seanmcilroy29 @jawache

[seanmcilroy29]

MoM

Chris opens the meeting at 1700 BST

Sustainable AI development and deployment.
Chris discussed AI carbon efficiency at the LF Energy Foundation AI dev conference. Responsible AI focuses on minimizing environmental impact through ethical and sustainable AI development and deployment. Asim discussed aligning "green AI" terminology to promote sustainability in AI development.

Standardizing AI sustainability framework.
The SCER framework is designed to standardize energy efficiency labelling for AI models. Chris discussed a certification program for LLMs, which includes two levels of participation: internal use and display on the website. There is also advocacy for the environmental impact of AI, focusing on categorization, benchmarking, and reporting usage. Chris proposes a framework for GreenAI certification.

Using impact framework to calculate SCI scores and avoid greenwashing.
Asim suggests minor things with "AI" in the title have been rejected, so it's important to ensure consensus and be prepared for potential backlash. Asim shared his screen to demonstrate a new project called Impact Framework, which calculates SEI scores. He explained that the Impact Framework is for computing and publishing green claims, addressing concerns of transparency and greenwashing.

Using blockchain for sustainability impact measurement and reporting.
Asim and Chris discuss ways to represent emissions reports on the blockchain.
Speakers discuss promoting JS work by cross-pollinating projects and connecting with other communities.
Asim: Proposed to use impact framework for SCI score calculation.

AI benchmarking for energy efficiency in the airline industry.
Speakers discuss complex issues in AI model development, focusing on common denominators and agreement among leading players. Chris proposes using hugging and publishing energy information to compete with carbon credits.
Asim and Chris discuss the need for globally recognized AI benchmarks for energy efficiency, such as those used by the airline industry. Sean emphasizes the importance of ecosystem discussion and stakeholder involvement in developing these benchmarks.

AI certification program for LLMs, feedback sought.
Sean O and Asim discussed incorporating information into a presentation to demonstrate MLM uses. Sean McIlroy and Sean discuss standardizing AI framework for industry adoption.

Creating a benchmark for a new operating system.
Chris discusses CPU performance and workload with Sandro. The group discussed creating a shared framework for open-source software development.

Action Items

• Chris - Incorporate suggestions around terminology and framing from Asim into the Share presentation
• Project members - Explore benchmarking approaches and how to prescribe standardised tests
• Sean - Schedule a follow-up discussion between Chris and Asim
• Chris - Share the presentation deck with GSF in advance of the June 15th conference
• Prepare Share framework documents for feedback at the June 2nd Standards Working

[jawache]

@sk16-dev this is generally ok for draft projects.

But we do need to start being more consistent and public regarding the state of this project, it's currently in draft which is an indication that it's something just in active discussion and it's not a spec we have consensus over. This is the explanation of the lifecycle of spec projects here: https://wiki.greensoftware.foundation/governance/projects/project-processes/project-lifecycle/document-project-lifecycle

@chrisxie-fw and @seanmcilroy29 I've created a pull request which is the standard disclaimer we add to all non-ratified projects here: #58

If you look at say the Impact Framework (which is in "incubation" state which is the software equivalent state of "draft" for spec projects) we are very clear in all the documentation and websites with a banner at the top like so https://if.greensoftware.org