Back in the day, when I was writing papers, it was a grueling, demanding process. I’d spend hours in the darkroom, trying to develop perfect exposures of all the images, and that was after weeks to months with my eyes locked to the microscope. Even worse was the writing; in those days we’d go through the paper word by word, checking every line for typos. We knew that once we submitted it, the reviewers would shred it and the gimlet-eyed editors would scrutinize it carefully before permitting our work to be typeset in the precious pages of their holy journal. It was serious work.
Nowadays, you just tell the computer to write the paper for you and say, fuck it.
That’s the message I get from this paper, Bridging the gap: explainable ai for autism diagnosis and parental support with TabPFNMix and SHAP, which was published in one of the Nature Publishing Group’s lesser journals, Nature Scientific Reports, an open-access outlet. Now I can’t follow the technical details because it’s so far outside my field, but it does declare right there in the title that they have an AI tool for autism diagnosis that is explainable, which implies to me that it generates diagnoses that would be comprehensible to families, right? This claim is also emphasized in the abstract, before it descends into jargon.
Autism Spectrum Disorder (ASD) is a complex neurodevelopmental condition that affects a growing number of individuals worldwide. Despite extensive research, the underlying causes of ASD remain largely unknown, with genetic predisposition, parental history, and environmental influences identified as potential risk factors. Diagnosing ASD remains challenging due to its highly variable presentation and overlap with other neurodevelopmental disorders. Early and accurate diagnosis is crucial for timely intervention, which can significantly improve developmental outcomes and parental support. This work presents a novel artificial intelligence (AI) and explainable AI (XAI)-based framework to enhance ASD diagnosis and provide interpretable insights for medical professionals and caregivers…
Great. That sounds like a worthy goal. I’d support that.
Deep in the paper, it explains that…
Keyes et al. critically examined the ethical implications of AI in autism diagnosis, emphasizing the dangers of dehumanizing narratives and the lack of attention to discursive harms in conventional AI ethics. They argued that AI systems must be transparent and interpretable to avoid perpetuating harmful stereotypes and to build trust among clinicians and caregivers.
So why is this Figure 1, the overall summary of the paper?
Overall working of the framework presented as an infographic.
You’d think someone, somewhere in the review pipeline, would have noticed that “runctitional,” “frymbiai,” and “Fexcectorn” aren’t even English words, that the charts are meaningless and unlabeled, that there is a multicolored brain floating at the top left, and that “AUTISM” is illustrated with a bicycle, for some reason? I can’t imagine handing this “explanatory” illustration to a caregiver and seeing the light of comprehension lighting up their eyes, which don’t exist in the faceless figure in the diagram, and perhaps she is more concerned with how her lower limbs have punched through the examining table.
This paper was presumably reviewed. The journal does have instructions for reviewers. There are rules about how reviewers can use AI tools.
Peer reviewers play a vital role in scientific publishing. Their expert evaluations and recommendations guide editors in their decisions and ensure that published research is valid, rigorous, and credible. Editors select peer reviewers primarily because of their in-depth knowledge of the subject matter or methods of the work they are asked to evaluate. This expertise is invaluable and irreplaceable. Peer reviewers are accountable for the accuracy and views expressed in their reports, and the peer review process operates on a principle of mutual trust between authors, reviewers and editors. Despite rapid progress, generative AI tools have considerable limitations: they can lack up-to-date knowledge and may produce nonsensical, biased or false information. Manuscripts may also include sensitive or proprietary information that should not be shared outside the peer review process. For these reasons we ask that, while Springer Nature explores providing our peer reviewers with access to safe AI tools, peer reviewers do not upload manuscripts into generative AI tools.
If any part of the evaluation of the claims made in the manuscript was in any way supported by an AI tool, we ask peer reviewers to declare the use of such tools transparently in the peer review report.
Clearly, those rules don’t apply to authors.
Also, unstated is the overall principle to be used by reviewers: just say, “aww, fuck it” and rubber-stamp your approval.
The first sentence is arguably wrong.
It’s thought that the actual accurence of ASDs is about the same as it always was.
What has changed is the frequency of diagnosis. We are much more aware of Austism Spectrum Disorders and much more aware that it is a spectrum, from Asperger like conditions to the more obvious autism like conditions.
The second sentence is also wrong.
We’ve known the causes of ASDs for decades now.
It’s mostly genetics with a high heritability of 80% or so and polygenic. It is also set during fetal development.
So, OK, the first and second sentences are wrong.
This is not a promising start for a paper.
If Nature (the publisher) wants to retain any scrap of the prestige associated with Nature (the journal), I would expect this paper to be retracted immediately, various reforms to the review process announced, etc. Any sign of that?
As of now, this notice appears right before the paper:
This at least is more or less accurate.
Each case of an ASD seems to be different in detail.
“If you’ve seen one case of an ASD, you’ve seen one case of an ASD.”
This probably reflects the underlying polygenic causes of ASDs. It’s thought that somewhere between 200 and 1,000 genes are involved, all with small effects acting in various combinations.
I work with explainable AI professionally. I skimmed the paper, and I have some questions.
The study is based on a sample size of only 120 (!!). They claim that the sample is “wide” rather than deep, meaning there are a lot of attributes on each patient. But… what are they then? The SHAP analysis only shows a handful, and I didn’t see discussion of aggregation method.
One of the problems with SHAP is that in the general case it’s very slow. It’s O(N!) in the number of attributes, intractable for a “wide” dataset. There are more efficient ways to calculate SHAP for specific models, but I have no idea if there exists one for the particular neural network they’re using. It stood out to me that when discussing computational efficiency, they left SHAP out.
Yikes, what a figure… it looks like someone took some popular AI related terms (F1-score, AUC, feature extraction, functional features, ReLu, Dropout, validation, test) put them through a meat grinder and then threw them at a wall to see where they would stick.
Also the x-axis on the figure bottom left.
I believe “explainable AI” means that, as with a human diagnostician, the AI can give reasons why it came up with that answer instead of some other, as opposed to “It’s because of a bunch of weights in the neural network, that were caused by the training data, but damned if anyone can figure out specifically how all that contributed to the current output, which means there’s no rational way to argue with it”.
I fexcectorn my frymbiai and now it’s not runctitional.
I learned in a previous AI figure thing that in some journals the figures don’t get reviewed.
Which blew my goddamned mind.
Correction to #5, that should be O(2^N) not O(N!).
stevewatson: generally, explainable systems are prediction systems that we can explain why a certain output came out, and more generally, we can model the model — i.e. the model is a complicated approximation of reality, too complicated for humans. But we can make a simpler model that a human can understand, that approximates what the complicated model is going to do.
This is particularly useful for understanding the circumstances where the complicated model is going to fail. This is useful when using the model, so you know whether it’s trustworthy. It’s also useful when doing research to come up with promising avenues for improving the model.
The concept is particularly applied to neural networks because they seem completely opaque, but it applies to modelling generally.
What’s going on with the legs of the adult in that illustration?
There’s already an expression of concern on it. The reviewers and editors may have failed, but the community at large came through!