Unfortunately, the “AI+Society” beat has started to bleed into my growing extracurricular interest in literature, because it is becoming increasingly obvious to me that everyone is illiterate now and it’s at least partly because of the chatbots. I’m not really the best person to lay out all the evidence from the education and publishing sectors supporting this claim, but once you realize it, you’ll see it everywhere. “Booktokers” — influencers whose job is, ostensibly, to read — complain about novels that have too many words. Anonymous, unedited Substack posts riddled with typos are circulated by pseudo-intellectual tech bros like the word of God. Don’t get me started on the rising popularity of the “video essay.”

And, if my most recent conference reviewing experiences are indicative of the general state of the field, today’s computer scientists have completely lost the ability to string two sentences together. The few papers I reviewed that were attempting to communicate the result of what seemed like sound and interesting research were shockingly incoherent. If they hadn’t attached code and figures proving that someone on the research team knew how to use a GPU, I’d find myself wondering whether some of these authors were overly ambitious fifth graders. You’d think that a reasonably educated person with underdeveloped language skills would at least think to work in a word processor with a built-in spelling or grammar checker, but alas, I found no evidence of this behavior.
 
It really just seems like researchers can’t (or won’t) read carefully anymore. Some authors failed to identify major relevant literature from the proceedings of the very conference they were submitting to. Others simply didn’t bother to add citations to work they described directly and referred to by name. Even some of the other conference reviewers spoke in platitudes rather than point out specific parts of the paper that exemplified the points they were making. (The submissions have line numbers! They are in red!!)

Most conferences have a policy against using AI in both submissions and reviews, which is a good thing for several unrelated ethical reasons. But if the authors can’t read or write, and the reviewers aren’t getting paid to proofread and edit these terrible drafts, then who’s flying the plane? Junior scholars are rarely formally trained in scientific writing, so a certain amount of this problem can be traced to gaps in access to mentorship and other resources, which is especially unfortunate. Still, I don’t exactly feel bad for gatekeeping publication status from researchers who can’t write — yet I don’t enjoy the process, either. Nobody wants to spend their time trying to parse nonsense cobbled together by strangers.

Maybe I’m being overly harsh. I empathize with everyone struggling with their reading practice in 2025. I took a long hiatus from reading, starting somewhere around my sophomore year of college (I majored in math) until, well, the summer after I finished my PhD in computer science. Yes, I had to “read” research papers and books to get through my degrees, but any academic will tell you that power-skimming for information with a deadline is a muscle that can be overtrained, and ultimately weakens your ability to actually engage with a text — sometimes resulting in a negative emotional association with the activity of reading itself. This was something that took time and effort to recover from, and I still think I would probably fail a serious college English class, if only because it takes me too much time to get through any book that could even marginally be described as “difficult.”

Too many people have thus decided that literacy is just not a personal priority in the age of chatbots and audiobooks. I think this is going to be the first (and possibly only) “existential” threat that AI poses to society. The silver lining for me is that if reading comprehension really does become as rare and coveted a skill as I fear it will, maybe I’ll escape the 2020’s with my job security intact. Anybody need a scribe?

So says OpenAI, the Microsoft-backed company behind the curtain of this creative entity, which is now in version 2.0. OpenAI plans to release the model as a tool for artists to quickly mock up their own ideas and experiment with different variations of their vision. However, like many recent advances in AI, DALL-E 2 has been met with criticism and concern. The model seems to replicate stereotypes and biases it learned from the data it was trained on: it shows users male “doctors” and female “nurses,” heterosexual “weddings,” and Western “building interiors” by default. Others have highlighted the tool’s potential to be abused to generate “fake news” and accelerate the downfall of democracy. Some have gone so far as to accuse DALL-E of being too smart: so effective and so creative that it will soon render artists—and art itself—obsolete.

Some of these concerns should be taken seriously. But we must be mindful of which ones we focus our collective attention on, lest we play into Big Tech’s hands. For even critical narratives can feed—and feed on—the AI hype machine that keeps them in business. And this hype machine is full of false promises of human-level intelligence.

Machine learning, the data-driven force behind the last decade’s colossal advances in AI, is not magic. As its name suggests, the underlying principle of this technology is the idea that we can “teach” an algorithm how to predict the future through exposure to the past. With the explosion in available data and processing power available to Silicon Valley’s most powerful players, modern algorithms can get a crash-course in a wide range of problems at a massive scale; DALL-E was forced to “study” over 650 million images as part of its training.

This anthropomorphic vision of AI, however, lends itself to exaggeration, and it is ultimately an imprecise metaphor. So-called “neural networks,” the high-dimensional functions that have made such good use of the computing resources Big Tech has given them, are not a real model of how our own nervous systems work. They are not sentient, and they are not autonomous: the humans building the technology are the real brains behind the operation. Engineers and researchers—real people—provide the tools the machine has at its disposal to build its solution, control the flow of curated information that allows it to self-evaluate and improve, and hand-pick the designs which perform the best according to their own judgment.

DALL-E 2, for example, is a particular Frankenstein of algorithmic parts that each do something precise and were chosen for specific reasons. One part is a function that encodes the user’s query into an image-caption pair; this part is built using Contrastive Language-Image Pre-training, a method which has been perfected and fine-tuned by many different researchers in recent years. The other two parts are also translation functions, but they belong to a family of efficient statistical functions called diffusion models, optimized jointly so that they play well together. Each optimization procedure is conducted and evaluated with data chosen specifically for this task.

You need not understand any of that for it to illustrate my point: DALL-E 2, like other state-of-the-art models, is a well-oiled machine hand-crafted to do its job (visual translation). It cannot find humor in a surprising juxtaposition, tap into the cultural zeitgeist, or communicate a novel idea—these tasks are, and always will be, best left to real artists. While DALL-E’s technical abilities are startling, art has never (just) been about accuracy—just ask anyone who continued to produce paintings when the camera was invented.

There is still so much scientists don’t know about how human intelligence actually works. For this reason, it’s difficult to prove that a neural network cannot exhibit properties like “creativity” by someone’s definition. But that doesn’t mean we need to worry about the collapse of human cultural institutions. If you want to worry about AI, worry about the actions of those who build and use it.

What are your favorite papers or authors on this topic? @ me next time!

Please stop permuting features

Hooker, G. and Mentch, L.

Such a polite alternative to “feature importance considered harmful.” “Wait,” you say. “Feature importance is not the same as XAI.” First of all, that acronym is awful. Second,

According to this paper, Leo Breiman committed an original sin when he invented random forests and started the trend of running partially scrambled data through a trained model to simulate and thus measure the effect of “removing” information.

Why was this such a bad idea? The short answer is, when you create semisynthetic instances that are actually very improbable in the true data distribution (and yes, many “explanation” methods are guilty of doing this too), your poor model is forced to extrapolate to a part of the feature space it has never seen, and that model’s extrapolation behavior to this new area is not typically relevant to whatever question you’re asking about the model.

This begs the question: What are you asking about the model anyway? And what information do you think the answer is giving you?? Examples of the questionable usage of feature importance for things other than feature selection abound in the following paper.

Trust in data science: collaboration, translation, and accountability in corporate data science projects

Passi, S. and Jackson, S.

If, like me, you have worked as a data scientist professionally, you will feel Extremely Seen by this paper. (I mean, probably. It’s a case study, so generalizability is not the point. I don’t care if you work at Amazing Organization Inc. and can’t relate.)

This paper is not just about explanations, as the title suggests, but it describes the very complex life cycle of a data science project including how the scientists negotiate confidence and trust of, for instance, management, or customers. Hint: it’s not just accuracy metrics! There is lots of explaining to be done.

There are some amazing quotes about the frequent use of intuition along with feature importance in model evaluation. I’m going to make you actually read the paper to get to them, but here is a great sentence very roughly summarizing some of the methodology described at the company investigated:

Often, intuitive results are made visible to inspire trust in models, while sometimes counter-intuitive results are posited as moments of new discoveries.

Okay, so if a feature with an “intuitive” connection to a model’s target appears to be the most predictive, that’s good, but if an “unintuitive” feature also turns out to be important, that’s …. also good? Did someone say confirmation bias? No? Well, this next paper does.

The intuitive appeal of explainable machines

Selbst, A. and Barocas, S.

Why are people so gung-ho about explanations in AI anyway? This paper analyzes the general desire for explanations in the air, arguing that it is because machine learning models can be “both inscrutable and nonintuitive,” therefore making it hard to tell whether a machine-made decision was justified from a legal or ethical standpoint. However, legislating to solve the scrutability problem (by requiring explanations, for instance) will not necessarily ease the task of testing for ethical properties:

In most cases, intuition serves as the unacknowledged bridge between a descriptive account and a normative evaluation.

What’s so wrong with intuition? It comes in handy so often! But it can also mislead, and the section of the paper on this point is my favorite part. Lots of references to “Thinking Fast and Slow,” which now I am thinking I should probably read.

Explanation in artificial intelligence: insights from the social sciences

Miller, T.

So, turns out this entire field is complicated by the fact that explanation in itself is a super confusing concept, but luckily this has also been studied for like, decades. This paper is a massive survey and analysis of existing literature on how humans generate, select, evaluate, and communicate explanatory information. Spoiler alert: causality is a pretty important part of the picture — but, as many have argued over the years, there’s a lot more going on. Here are what I think are the 2 most important findings in the paper, from my own biased perspective:

The punchline once you get into the weeds, of course, is that a lot of so-called “XAI” solutions are far, far removed from what humans actually desire from an explanation.

What this paper doesn’t do is present a legal, methodological, or moral necessity for explanation; instead, it acknowledges the fact that “the running hypothesis is that by building more transparent, interpretable, or explainable systems, users will be better equipped to understand and therefore trust the intelligent agents.”

Towards a rigorous science of interpretable machine learning

Doshi-Velez, F. and Kim, B.

I, for one, am fully on board with equipping users to understand and “normatively evaluate” the algorithmic systems they engage with. But methodologically sound experiments must be done to show this is actually what explanation methods can achieve, because I think we have reason to be skeptical at this point.

So I will leave you with this final paper. Happy researching, folx. Might want to brush up on your HCI.

My Netflix recommendations suck because I often binge watch “addictive” shows I eventually regret spending time on. My Amazon recommendations suck because I only use it to buy expensive textbooks that I desperately need. My Twitter recommendations sucked for a while because I obsessively read the profiles of people I hated. I will admit that Instagram had me figured out at one point in that it only showed me items of clothing in a 70’s burnt orange and I clicked on every single ad. Now it just shows me products related to my last Google search. (Please tell me how to get it to go back to orange stuff.)

I am being unfair here–I know that these algorithms are working the way they are supposed to–I just have to actually start using these services as they are intended to be used, and feed them more information about myself (haha, no). But I also have this sense that whatever it is that I like about stuff is not, fundamentally, a latent property of the stuff in question, or even a latent property of me as a consumer!

My reactions to music, for instance, are very subjective. Sure, there are certain genres and sonic features that I tend to like or dislike (thank God the Thrift-Shop-inspired Pop Sax Hook trend has died). But when I hear a song it also makes me think of… where I was when I first heard the song, what I know about who performed the song, what the lyrics remind me of… that kind of thing. Lots of context contributes to how I feel about the song, and yet it has nothing to do with the song–nor is it something that I would necessarily share with listeners similar to myself.

Not only does the “context” bit make what I like difficult to predict, but the process by which content was recommended to me actually affects how I feel about the content! I was born just in time to have experienced Mixtape Culture in high school and I still remember everyone who gifted me music (because I keep track, using iTunes playlists). I once dated a guy who was into metal. The breakup was awful, and I get incredibly angry every time I hear one of the songs that he gave me, so I listen to them when I go to the gym. Imagine if a recommender system had suggested that music to me instead (it wouldn’t have). I wouldn’t have the same relationship with it!

I was at a great talk by Nick Seaver about interpretability in music recommendation the other day which partly went over my head (he is an anthropologist). Something he suggested about how data scientists think about music recommendation as a problem stuck out to me: it is driven by a fundamental anxiety about the vastness of Content and the resulting desire to maximize and optimize exposure to that Content. Sad! I do not share this anxiety. Content consumption is one of those human-y things that I tend to like to do the old-fashioned way.

This one goes out to the content man.

The idea for the solution comes from the failure of many fairness metrics under Simpson’s paradox, where a signal may appear different, or disappear completely, under different grouping conditions. Basically, given a sensitive input X and outcome of interest Y, as well as “admissible” inputs S (determined perhaps by a social scientist or domain expert familiar with how the data was collected), X should not “affect” (don’t ask me to define this) Y no matter which subset of features S are conditioned on. Now, once you have a dataset that “minimally” satisfies this property in some way (think: adding and removing some tuples), you have something that should resemble your original dataset, but without those pesky “encoded historical biases” everyone has been worried about so much.

I think this work is great for lots of reasons. I wonder, though, how people should interpret their analyses of the new dataset. If we know that historical outcome Y is biased in a certain way, what does it mean to build a model that accurately recovers Y values given the admissible features in this “corrected” setting? Let’s say that we were interested in evaluating job aptitude, and we are using historical promotion data to decide whether to hire people. The fundamental problem here is that promotion != aptitude, because, perhaps among other reasons, women are promoted less frequently by sexist superiors, or don’t ask for raises because of social conditioning. So we correct the data for historical biases against women. Our new target is still promotion, but a model we train on the admissible features should be less sexist (by a number of metrics, I think- see the forthcoming papers) because the data now satisfies the assertion that gender has nothing to do with aptitude.

But how do we know the signal in this new data is actually a better proxy for aptitude? If optimizing for promotion was such a terrible idea in the first place, what exactly is left that would make it a good idea now? Even in the case where domain experts are telling you what to control on, who makes the call on whether the data could possibly contain any meaningful information? A related question I didn’t get to ask Bill is whether this data would be useful for scientific inquiry / causal inference in the many settings where a prediction isn’t actually what you want but I don’t know anything about causal inference anyway so maybe someone who does can chat with him about it.

I went over the current state of research on local explanation methods and how we are starting to use SHAP values as a jumping-off point for learning about & debugging our model, explaining results to stakeholders across the company, and providing information to applicants.