Direkt zum Hauptbereich

When Compiler Engineers Act As Judges, What Can Possibly Go Wrong? How LLVM's CoC Committee Violated Its Own Code

Open source thrives on collaboration. Users report bugs, developers investigate, and together, the software ecosystem improves. However, the interactions are not always trouble free. Central to this ecosystem are Codes of Conduct (CoCs), designed to ensure respectful interactions and provide a mechanism for addressing behavior that undermines collaboration. These CoCs and their enforcement are often a hotly disputed topic. Rightfully so! What happens when the CoC process itself appears to fail, seemingly protecting established contributors while penalizing those who report issues? As both a law professional with a rich experience in academia and practice as a legal expert who also contributes to various open source software projects over the past couple of years, I deeply care about what the open source community can learn from the law and its professional interpreters. This story hopefully ignites the urge to come up with better procedures that improve the quality of conflict resolution outcomes.

And there is much to learn both ways! This is the story of my personal experience with the LLVM project in GitHub Issue #72413, a journey that started with a simple bug report for a harmless but annoying issue and ended with a Code of Conduct Committee decision that I believe profoundly failed in its duty to ensure fairness and accountability and therefore damages the reputation of the LLVM project as a whole.

The Spark: Reporting a Valid Bug

In November 2023, I opened Issue #72413 reporting numerous warning messages during builds of LLVM's compiler-rt component. As an end-user (specifically, someone involved with CachyOS who compiles a lot of open source software using LLVM/Clang with highly optimized PKGBUILDs), my goal was simply to report a regression I observed.

The initial response from Gentoo developer and LLVM contributor Sam James (thesamesam) was unfortunately dismissive, suggesting it was "likely your toolchain setting it or your build script". This premature conclusion, later proven incorrect when another user (mati865) identified the causative commit, set an unhelpful tone from the start.

Providing Proof Amidst Challenges

Undeterred, and after the issue was confirmed by others, I dedicated significant effort in early February 2025 to provide comprehensive information for developers. This included:

  • Detailed logs (PKGBUILD, makepkg.conf, CMake logs).

  • Scripts that reproduce the problem.

  • Testing under different build configurations (ENABLE_PROJECTS vs. ENABLE_RUNTIMES).

Thankfully, developer Alexander Richardson (arichardson) who was responsible for the code change that introduced the regression and some other users engaged constructively, leading to a potential fix being developed. This is how the process should work!

The Turn: Unwarranted Hostility and Escalation

However, the situation deteriorated sharply when Sam James refused to engage with the provided reproducer scripts by abruptly stating "I'll unsubscribe from this bug now", abandoning the issue. I was deeply fed up by then for his unwillingness to come up with any constructive and forthcoming contributions of his own in that thread to solve the technical issue. While I had already put in a significant amount of time and effort, he still kept demanding for more but wasn't showing good will to invest mere seconds for trivial changes to my scripts. Maybe the comment that voiced my anger crossed a line, too. I take full responsibility for that. But I think that this provoked reaction is understandable after all the time and effort spent to solve this issue constructively by a non-technical person. I did soften my tone in the follow-up comments [1], [2], significantly to show my willingness to de-escalate and focus on the technical issues instead and came up with a proposed fix made with AI later on. While that turned out to be flawed, it further demonstrated my good intentions to contribute to a solution for a bug.

The intervention of yet another Gentoo developer, Eli Schwartz (eli-schwartz), brought the drama to a whole new level. Instead of focusing on the technical issue, Schwartz launched into a series of comments that were, in my view, sarcastic, dismissive, personally insulting, and escalated the situation unnecessarily:

This is not what I consider to be tolerable behavior.

The CoC Process: Seeking Accountability, Finding Failure

Facing this hostility, and after attempting de-escalation myself, I felt I had no choice but to invoke the process designed for such situations: I filed a formal report with the LLVM Code of Conduct Committee. I documented the behavior I found unacceptable, hoping for a fair review.

The committee's decision, delivered on April 9th, 2025, was shocking. It concluded that Eli Schwartz and Sam James did not violate the Code of Conduct. Instead, it found me in violation, citing principles like "be considerate" and "be kind," based on wrong interpretations of my comments made during the heated exchange initiated by others. They encouraged me to apologize.

Deconstructing the Committee's Flawed Decision

The committee's judgment is, in my firm opinion, deeply flawed and fails to withstand scrutiny when compared against the documented record:

  1. Ignoring Blatant Violations: The committee completely absolved Eli Schwartz, whose comments contained direct personal insults, sarcasm, and inflammatory remarks – clear violations of the CoC's core tenets of respectful communication. Sam James's dismissiveness and disengagement were also ignored.

  2. Mischaracterizing My Actions:

    • They claimed I was "Expecting a solution and refusing to provide the help..." This is factually incorrect. I provided extensive diagnostics and even an analysis and tested code modification (even if imperfect, it showed effort and moved towards resolution). The linked comment shows me offering help.

    • They focused on my frustrated phrase "actually do your job" while ignoring the preceding hostility, the context that provoked it and the rest of that particular comment that provided further context for my frustrations. My phrasing was harsh, yes, but it wasn't an unprovoked attack like those I received. Having to change a few lines of code in my provided scripts is not too much to ask for a developer for reproducing the issue. Not for the involved Gentoo developers, I suppose.

    • They accused me of "Making threats and weaponizing the Code of Conduct" by linking to Eli Schwartz's comment. Stating I had filed a report (my Feb 5 comment) after enduring abuse is using the prescribed process, not weaponizing it. So much for a fair assessment of the facts in that thread.

    • They claimed I implied "[my] time is more valuable](https://github.com/llvm/llvm-project/issues/72413#issuecomment-2632427578)" based on Eli's hostile misreading of a common phrase about limited resources (my Feb 4 comment). They even linked this finding incorrectly to the "do your job" comment.

  3. Ignoring Context and Timeline: The committee failed to recognize that Eli Schwartz's aggression preceded my stronger reactions. His behavior was the catalyst for escalation, not the other way around. Their judgment twists cause and effect.

  4. Overlooking Positive Contributions: The decision completely ignores that despite the hostility, I persisted and contributed analysis and code changes that directly addressed the technical issue. This contradicts any narrative that I was merely complaining without contributing.

  5. Apparent Bias and Lack of Accountability: The outcome strongly suggests a bias favoring established contributors over external users. It magnified my reactive missteps while minimizing or ignoring clear, proactive CoC violations by others. This isn't impartial judgment; it's a failure of accountability.

My Stance and Broader Implications

I have formally rebutted the committee's decision and stated that I cannot accept their flawed judgment. This isn't just about one minor code issue any longer; it's about the integrity of the Code of Conduct process and the health of the LLVM community.

When a CoC committee fails so clearly, it sends a chilling message:

  • Users reporting bugs may face hostility with little recourse.

  • Established contributors can seemingly act with impunity.

  • Non-technical contributions (like detailed reporting, diagnostics, and even analysis assistance) are undervalued or dismissed.

  • The process meant to ensure a welcoming environment can be used to silence those who challenge unacceptable behavior from established contributors.

This LLVM CoC committee's handling of Issue #72413 represents a failure to act responsibly and impartially. It undermines trust in the very mechanisms designed to foster a healthy, collaborative community. Open source depends on contributions from everyone, and that requires CoC processes that are fair, thorough, unbiased, and hold everyone accountable to the same standards. In this case, LLVM's process fell disturbingly short.

The broader community deserves to know how these situations are handled, I think. It's time for a conversation about how to ensure Code of Conduct committees actually fulfill their crucial role as judges in such conflicts effectively and equitably, ensuring that open source remains truly open and welcoming to all who wish to contribute in good faith. This example highlights the harm that can be done to a community if important members fail to live up to their own standards.

The Appeal Process Begins: Further Concerns Arise

Following my detailed rebuttal outlining the failures of the initial decision on April 9th 2025, I received a response from the LLVM CoC committee on April 10th 2025 acknowledging the appeals process. However, their communication immediately raised further concerns by cautioning me against using "threats and demands" – seemingly interpreting my deadline for confirmation and my stated intention to seek transparency as such.

This necessitated a further response to clarify my position and formally initiate the appeal while addressing these new points. Key arguments I raised in my formal appeal communication include:

1. I categorically rejected the interpretation of my actions as "threats" or "demands."

"Setting a deadline for confirmation that my appeal is being considered is a standard request for procedural clarity. Stating my intention to publicly share my documented experience with this process, should it lack a satisfactory resolution, is an exercise in seeking accountability, not an act of intimidation... My requests for reconsideration, acknowledgement of documented CoC violations by others, and a revised, fact-based assessment are substantive points of my appeal, not impermissible 'demands.'"

My intention was clearly stated as seeking accountability and transparency, leveraging the LLVM CoC's own definitions which limit actionable "threats" to contexts like physical danger or ongoing harassment – clearly not applicable here.

2. I argued that the initial decision itself constitutes a violation of the LLVM Code of Conduct by the committee members involved.

"A CoC process that results in a decision perceived as biased, dismissive of evidence, and disrespectful towards a participant fails to uphold the very spirit and letter of the Code of Conduct it is meant to enforce... Therefore, the decision itself, in its clear disregard for fairness, context, and the principles of respectful and considerate engagement, constitutes a failure to adhere to the LLVM Code of Conduct by its own administrators."

By ignoring evidence, demonstrating imbalance, and failing to adhere to principles like "be respectful," "be considerate," "be welcoming," and "try to understand why," the committee undermined the very foundation it is supposed to uphold.

3. Formal Request for Recusal Due to Bias: Given the flaws and apparent bias in the original decision, I formally requested the recusal of the original committee members from handling the appeal.

"Based on the significant and demonstrable flaws in the initial decision... I hereby formally request the recusal of all committee members who either participated in rendering that original decision or where there is a conflict of interest by being too close to the Gentoo project from any involvement in this appeal process... Expecting the same members who produced a decision exhibiting such clear analytical flaws and apparent biases to now conduct an objective review of a challenge to that very decision is contrary to basic principles of procedural fairness."

I also explicitly raised concerns about potential conflicts of interest stemming from closeness to the Gentoo project, given the individuals involved in the original issue.

4. Highlighting Procedural Vagueness: I noted the lack of clarity in the documented appeals process regarding safeguards for impartiality.

"Please confirm that this procedural safeguard [review by uninvolved members] will be implemented for the handling of my appeal as it isn't clearly stated in the appeal process section of the response guide. That vagueness of the appeals process is a point of concern... Without the stated safeguards in place, I would not consider the appeals process fair and sound."

The Saga Continues: Procedural Roadblocks and Cross-Project Conflict

Just when you think a process focused on conduct might prioritize substance, procedural maneuvering often takes center stage. Following my detailed appeal, outlining the significant flaws in their initial decision, their response was unfortunately telling. Instead of engaging with the core arguments about ignored evidence, mischaracterizations, and apparent bias, the committee replied asking only for "new or different evidence," stating the appeal "looks to only refer to the public GitHub issue report that was already analyzed."

This response is a classic example of procedural obstruction. It attempts to invalidate an appeal based on flawed reasoning by demanding new facts. My appeal's central argument is that their initial analysis of the existing facts was fundamentally flawed and biased. To demand entirely new evidence as the sole basis for appeal effectively shields the original, faulty decision-making process from scrutiny. It's a refusal to engage with the substance of the critique, raising serious questions about the committee's willingness to correct its own errors or act in good faith. I have formally responded, reiterating that a flawed analysis is grounds for appeal and restating my request for recusal of the original members.

This latest exchange underscores the ongoing challenges in achieving a fair and impartial review within the current LLVM CoC framework. It highlights a reluctance to engage with substantive criticism and a tendency to control the narrative rather than addressing the core issues of accountability and respectful community engagement raised by the original incident.

As the law is my profession, questions of accountability and fairness matter deeply to me, it should be a concern to all LLVM contributors as I call the integrity of the process and the involved committee members into question. As far as I see, none of the committee members bring some kind of legal experience to the table. They might be distinguished compiler engineers, but as this case has demonstrated, that might not qualify them as a good judge. I have confronted the LLVM Board of Directors with this case and asked the Board to take action on my behalf to correct the presented shortcomings, but in their reply from May 10th, the Board has concluded that they agree with the decision and resolution made by the LLVM Code of Conduct Committee. 

Hence I am forced to take this case to the public to expose all of these shortcomings as the people involved were not able to act upon their own mistakes responsibly and damaged the LLVM community to a great amount.

If you want to know who the current committee members are or who sits on the Board of Directors, just follow the given links.

Conflict Spills Over: The Mesa Incident

Disturbingly, the negativity surrounding the LLVM CoC issue wasn't contained. Shortly after these events, I was involved in a separate incident on the Mesa project's GitLab (Mesa Issue #13022). After providing a detailed bug report (which included a brief AI analysis at first that turned out to be controversial for some), bisect, and testing patches that led to a swift technical resolution by the Mesa developers (kudos to Timur Kristóf!), the discussion took a negative turn, initiated by Matt Turner who works for Google but is affiliated with Gentoo.

Despite having already removed a section of AI-assisted analysis from my original Mesa post at the first suggestion it wasn't helpful, Matt Turner explicitly linked the LLVM CoC controversy into the Mesa thread, characterizing my use of AI there as "noise" and "net negative."

This opened the door for Eli Schwartz to once again inject personal hostility. He entered the Mesa thread repeating accusations derived from the LLVM conflict, labeling my good-faith (though admittedly imperfect) attempt to use AI as "harassment by LLMs," employing sarcastic analogies, and falsely claiming I had accused LLVM developers of "failing in their fiduciary duties." He explicitly stated my actions were "not good faith."

I must respectfully disagree with the framing that implies my actions inherently showed a lack of respect or unduly imposed upon others in this specific Mesa instance or necessarily justified the importation of external conflicts. My use of an AI tool, for example, was explicitly stated as an attempt to bridge a knowledge gap and assist diagnosis, offered in good faith, not as a finished product demanding review or intending disrespect. That it provoked such strong reactions yet again from some developers was surprising to me. The part in the original OP was also very brief. While its utility can certainly be debated, framing its mere use as inherently disrespectful feels overly harsh. There is also no Mesa policy known to me that governs the use of such AI analysis. The Gentoo developers pointed me towards their clear policy, but from the comments of other developers received in the thread, some take a more pragmatic approach and I take away that there is no consensus on this topic for the Mesa project. Yet I faced some hostile reactions from different developers even though I proactively acted upon the initial reaction from Timur and edited the OP to delete the AI analysis. The other part of AI analysis that remained in the thread was a summary of my downstream changes in the hope that they contained something useful for upstream. My wording made it clear that I left it entirely to the developers to decide if they want to spend some time on it or not. They chose to take a brief look and gave me useful comments as I cannot judge the technical merits of the generated code and analysis on my own. I am grateful for such constructive criticism. However, everything from the point when Matt Turner entered the thread, I consider destructive as I cannot see anything useful in them for the already solved technical issue, it was solely meant to discredit me personally.

Once again these two Gentoo developers showed a lack of good manners. This cross-project importation of personal grievances and hostility is unacceptable. It derailed the Mesa discussion, ultimately leading to the Mesa maintainer locking the thread. It demonstrates a pattern of behavior that extends beyond a single project and appears targeted. While the Mesa maintainers acted appropriately by locking the thread to stop the derailment, the incident serves as further evidence of the unproductive and hostile engagement non-technical contributors can face, initiated and perpetuated by individuals that might hold a personal grudge against me.

These latest developments – the LLVM CoC committee's procedural deflection and the spillover of hostility into the Mesa project – further underscore the systemic issues at play in the FOSS ecosystem. It highlights not only a potentially broken internal review process within LLVM but also a willingness by some individuals to carry conflicts across community boundaries, poisoning interactions elsewhere. It strengthens the case that transparency and accountability are urgently needed.

To end on a positive note, the other kind of developer deserves my respect that remains calm under stress, shows respect to others, comes up with pragmatic solutions and can keep their personal emotions under control when interacting with non-technical people like me. I can imagine that from their perspective, it isn't always easy to interact with laymen and the sheer amount of issue reports can be taxing. I value such professional behavior very much - you know who you are.

Beliebte Posts aus diesem Blog

Linux Gaming Tweaks - A small guide to unlock more performance (1)

My personal journey to unlock more performance on Linux - Part 1: Introduction This is the start of a new series dedicated to the Linux Gaming community. This is a bit of an oddball in my blog as most of my other blog posts are written for a German audience and cover my other two passions: politics and the law. Nonetheless, PC gaming is a hobby for me since I was six years old, playing games on a Schneider 386 SX. Wow, times ran fast. As I've learned quite a lot about Linux during the last couple of years, switching between several distributions, learning about compilers and optimizing parts of a Linux distribution for a greater gaming experience, I was asked recently on the Phoronix Forums to share some of my findings publicly, and I am very glad to do so with a global audience. But keep in mind, I am neither a software nor a hardware engineer - I am a law professional who is passionate about computers. I digged deep into the documentation and compiled a lot of code, breaking my s...

Amtsschimmel - Folge 4 (Fortsetzung 3) - Die Generalstaatsanwaltschaft steckt den Kopf in den Sand

Wenn es um das Sühnen staatlichen Unrechts geht, ist in der Regel auf eines Verlass: Auf eine groteske Verweigerungshaltung anderer staatlicher Stellen dies anzuerkennen und in der Folge auch zu ahnden. Wer den Ausgangsfall verpasst hat, sollte unbedingt sich zuvor den Beitrag hier noch einmal anschauen. Widmen wir uns heute dem Bescheid der Generalstaatsanwaltschaft Rostock vom 10. Januar 2024 (Az.: 2 Zs 724/23), der inhaltlich bedauerlicherweise wieder einer Arbeitsverweigerung gleich kommt. Immerhin stellt man sich dabei leicht intelligenter an als  noch die Staatsanwaltschaft Schwerin , wenn auch im Ergebnis ohne Substanz: Lieber Kollege Henkelmann , haben Sie wirklich über die Jahre alles vergessen, was Sie einmal im Staatsrecht gehört haben sollten? So grundlegende Dinge, wie die Bindung aller staatlicher Gewalt an die Grundrechte (Art. 1 Abs. 3 GG) oder das Rechtsstaatsprinzip (Art. 20 Abs. 3 GG)?! Fühlen Sie sich auch noch gut dabei, wenn Sie tatkräftig dabei mithelfen, da...