A common way of generating a “hook” in a technical talk is to say “actually, this is really an old idea.” There are two examples of this that come to mind for me, group testing and randomized response. In both of these topics, there is a “classic paper” with an “interesting historical anecdote” that generates a kind of factoid in the audience’s mind. Unfortunately, the factoid that gets stored is often incorrect.

Group testing refers to the process of finding a (small) number of “defective elements” a larger set of elements by testing groups of elements. The assumption is that the test you have is sensitive enough to flag a group as containing a defective element. This was first proposed in Robert Dorfman’s 1943 paper The Detection of Defective Members of Large Populations in The Annals of Mathematical Statistics. He introduced group testing with the application of screening for syphilis in the United States Public Health Service and the Selective Service System during WWII. A syphilis test (the Wasserman test) which is sufficiently sensitive could be used on pooled blood samples: if the test was negative the whole group is clear and if the group is positive you could test individuals in the group or further subdivide.

As noted in this paper by Gilbert and Strauss, the “Selective Service System did not put group testing for syphilis into practice because the Wassermann tests were not sensitive enough…” when pooled. That paper doesn’t have a citation but you can find it in the book by Du and Hwang: 

Unfortunately, this very promising idea of grouping blood samples for syphilis screening was not actually put to use. The main reason, communicated to us by C. Eisenhart, was that the test was no longer accurate when as few as eight or nine samples were pooled. Nevertheless, test accuracy could have been improved over years, or possibly not a serious problem in screening for another disease. Therefore we quoted the above from Dorfman in length not only because of its historical significance, but also because at this age of a potential AIDS epidemic, Dorfman’s clear account of applying group testing to screen syphilitic individuals may have new impact to the medical world and the health service sector.

It seems that group testing was not used for the syphilis screenings. Most people are careful to say it was proposed and not used but without “closing the loop” people learning about it for the first time could be misled. Group testing has been used for many other diseases, most recently in some approaches for COVID screening.

The second example is randomized response, which is a technique for providing plausible deniability to survey respondents. A surveyer asks a sensitive binary question to an interviewee. The interviewee’s true answer is X∈{0,1}, samples a Bernoulli(p) random variable Z∈{0,1} (“flips a biased coin”) and responds with Y=X⊕Z where ⊕ is addition modulo 2. Randomized response was proposed by Stanley L. Warner in his 1965 JASA paper Randomized response: A survey technique for eliminating evasive answer bias. Talks on differential privacy (especially local differential privacy) often trot out Warner’s paper as an example of how differential privacy has appeared “classically.” I have done this myself.

Unfortunately, as a 2015 JASA paper of Blair, Imai, and Zhou notes:

Despite the wide applicability of the randomized response technique and the methodological advances, we find surprisingly few applications. Indeed, our extensive search yields only a handful of published studies that use the randomized response method to answer substantive questions… 

The earliest study they could find was by Madigan et al. from 1976, who looked at a Misamis Oriental province in northern Mindanao (Philippines) and the prevalance of hiding deaths from official counts. So it seems randomized response was not implemented for around a decade after being proposed.

These examples show how easy it is to create misunderstandings by using anecdnotes about prior work in talks. I am certainly guilty of both misunderstanding the actual facts and perhaps misrepresenting what actually happened after these cool ideas were first proposed. We all know that the gap between theory and practice can be large, but somehow these fun stories make us a bit less careful.

So if Canvas is so awful, shouldn’t I use something else? Maybe. It helps to imagine (with some dramatic liberties) the evolution of the course website:

• Everything’s on paper. There’s a book or lecture notes/a reader you buy from the bookstore or copy shop, assignments are physical handouts only (photocopies or dittos or something). Every class works this way. Scores for assignments have to be manually associated with students.
• Same book or lecture notes/reader but homework files are on the web (or ftp or gopher maybe) in .ps (or later .pdf). A course website is some hand-coded HTML (like my current homepage!) Students can maybe pick up a printout or print it themselves in a computer lab or at home. We can call this the “bag of PDFs” model. Scores for assignments have to be manually associated with students.
• The website design is somehow centrally controlled either via a template or something and now the book/notes are in .pdf but many students still print things out because lugging a laptop around feels annoying. Maybe a pretty bag of PDFs.
• The dawn of the LMS: student rosters can get associated into a system where you can deposit your bag of PDFs and then organize them in some pre-specified way. Grades for assignments can be manually associated with students in the system and then you can submit them automagically!
• The creep of the LMS: you can make quizzes/assignments (simple ones) that are auto graded, make your site look pretty, maybe embed some videos and provide other content, and generally automate some aspects of your class. To take advantage of “features” you have to change your class to meet the tools. The latter is appealing because some features help students learn better (at least according to some research). It’s an opportunity to try things new.
• Late-stage capitalism LMS: Universities “mandate” that faculty use a particular LMS. Many faculty comply. Every year new edtech companies show up trying to get you to use their software (and sometimes steal student data). Some might be grifts, others are not. They heavily marketed and different ones are pushed by different teaching and learning centers. Many of them require you to change your teaching to fit the tool because they are one-size-fits-all: claims to works for all fields, all types of classes!

So why am I still using Canvas? The main reason is that it benefits the students, not because it is good, but because they have been using it for 2 years and they are used to it. They can see all their due dates on a single dashboard. If you are reading this and scoffing, think about how terrible and non-interoperable almost every calendar system is. If you say “you’re just bowing to peer pressure” you’re more or less right. If you say “but it builds character for students to have to manage their due dates” then I’d ask if the goal of your class is to teach the material or to teach time management. If you say “both” then does your class explicitly teach students time management skills? I’m guessing not.

In this new class format, there are 28 class sessions. In 26 of them there is a conceptual quiz students have to take before class as well as an in-class assignment for which they have to upload solutions. Then there are homeworks, quizzes, and projects/labs. Compared to the old 8 problem sets, two midterms, and a final, that’s more than a 6x increase in things to keep to track of for > 200 students. It behooves me, as someone who cares whether students learn the material, to try and make keeping track easier.

Ultimately a university “adopting” any LMS is coercive because if students use it for all their introductory classes then using something else is almost deliberately making their lives harder with no real benefit. I don’t think I’m going to William F. Buckley it up and stand athwart with my bag of PDFs (even if it is on github). Ultimately I think using the LMS the right thing to do by the students. I’m going to be super salty about it though.

At the beginning of August I went to campus to look at the classroom, which is Lucy Stone Hall on the Livingston campus at Rutgers, which can seat 400 students. For context, to get there from my office/where most STEM classes are (on Busch campus) one has to drive, take a bus, or bike. I previously taught in a classroom that can seat 147 which had whiteboards on rollers that went up and down (they are behind the projection screen), which gives 4 boards worth of space visible at a time. The new classroom supposedly has “5 chalkboards” but those are on wooden panels that are partially obscured by the podium. The chalkboard on casters shown in the picture is not particularly visible from the back of the class. So… no real board space. I can use the projection screen with a tablet of document camera (or maybe transparencies to be really old school) but with a single projection screen.

So I’ve embarked on a far too ambitious plan to partially “flip” the class: students will watch video lectures (already recorded during COVID times) and then come to class to do more active learning/problem solving activities. Since this blog has been moribund for the last few years, I will try to write about this process as it goes to help process/document what I’m doing and how well its working (or not).

I’ve been doing a bit of reading on prior approaches, including:

• Van Veen’s 2013 paper on flipping DSP and his YouTube channel
• Fowler’s 2014 paper on flipping a large ECE class at SUNY Binghamton and his class website.
• Lygeros and Sutter’s 2016 paper on flipping a MechE signals and systems class.
• My own colleague Waheed Bajwa’s experience in flipping his DSP class (for which my class is a prereq)

I’ve also gotten a lot of help from various friends and other educators about their own experiences and ideas of what has worked and what hasn’t.

I rapidly realized that I could not implement all of these ideas in a month before the semester so I am trying to pick and choose my battles. Usually, to successfully flip a class requires a high instructor (TAs, learning assistants (LAs), etc.) to student ratio. I don’t even know how many TAs I’ll have this semester yet, so that’s going to be a challenge. It’s waaaaay too late to ask for LAs. Hence I’m calling this a “partial” flip.

I’m not sure I’ll be able to pull it off, but here’s hoping!

“The inferential statistical methods on which machine learning is based, while useful in many contexts, fail when applied to extrapolating subjective human characteristics from physical features and even patterns of behavior, just as phrenology and physiognomy did.”

From the (mathematical) communication theory context in which I teach these methods, they are indeed useful. But I should probably teach more about the (non-mathematical) limitations of those methods. Ultimately, even if I tell myself that I am teaching theory, that theory has a domain of application which is both mathematically and normatively constrained. We get trained in the former but not in the latter. Teaching a methodology without a discussion of its limitations is a bit like teaching someone how to shoot a gun without any discussion of safety [*]

The paper describes the parallels between the development of physiognomy and some AI-based computer vision applications to illustrate how claims about the utility or social good arguments made now are nearly identical. They quote Lorenzo Niles Fowler, a phrenologist: “All teachers would be more successful if, by the aid of Phrenology, they trained their pupils with reference to their mental capacities.” Compare this to the push for using ML to generate individual learning plans.

The problem is not (necessarily) that giving students individualized instruction is bad, but that ML’s “internally consistent, but largely self-referential epistemological framework” cherry picks what it wants from the application domain to find a nail for the ML hammer. As they write: “[s]uch justifications also often point to extant scientific literature from other fields, often without delving its details and effacing controversies and disagreements within the original discipline.”

Getting back to pedagogy, I think it’s important to address this “everything looks like a nail” phenomenon. One start is to think carefully even about the cartoon examples we use as examples. But perhaps I should add a supplemental reading list to go along with each topic. We fancy ourselves as theorists, but I think it’s a dodge. Students are taking the class because they want to learn ML because they are excited about doing machine learning. When they go off into industry, they should be able to think critically about whether the tool is right for the job: not just “is logistic loss the right loss function” but “is this even the right question to be asking or trying to answer?”

[*] That is, very American?

When preparing the course, problems for homeworks and quizzes (assessments) could be organized into Question Banks stored in the professor’s office.

• To create an assessment, a special place was prepared on the wall of the classroom.
• The professor would photocopy the Questions from the Question Bank, one for each student, and would post them in sheaves on the wall for the students to take.
• The students would collect the assessment. When they completed it, the scores would be tallied and magically stored in the Professor’s ledger.
• The go-getting-est among them would realize some error or typo on a Question and inform the professor.
  Even though the professor could alter the Question in the Question Bank, the photocopies had been made and he could not alter the Questions already taken by the students.
• However, he could amend the photocopies on the wall for those laggards who had not collected their assessment from the wall. For those students, the Question was corrected.
• The amended Question was only corrected on those assessments left on the wall. Harried for time, they (the professor) may or may not have corrected the Question in the Question Bank. This will be important later.
• The laggards would have correct tallies but the go-getters’ tallies had to be manually amended, one by one.

At the start of the next year, the professor, exhausted, asked “can I not reuse the materials from last year?” The high priests of Canvas exclaimed “but of course!” in a tone that suggested the professor was interested in mustard of Francophone origin. The professor chanted the strange rituals and lo and behold, a copy of the class was prepared.

For each section of wall (assessment), sheaves of amended Problems were already there, rustling in the autumnal breeze.

• The amended Problems did not appear in the Question Banks. The magic of Canvas was unable to “scan” a problem from the wall and copy it into a Question Bank. It was as if the Question on the assessment was an entirely different object from the Question in the Question Bank.
• The professor, full of great plans for improved assessments, began to amend the Question Banks. However, since photocopies had been made, so the professor tore down the sheaves of Problems on the wall and prepared to make a fresh batch of photocopies.
• However, the magic of Canvas had also created phantom duplicates of the Question Banks in the professor’s office, so the amended Questions and unamended Questions were hard to distinguish. Moreover, problems with editable equations were copied as uneditable photographs (PNGs).

The professor, distraught, turned to their local priest. The priest, unable to help, conveyed the question to higher authorities, where it vanished.

Meanwhile, the professor’s assistants waited for some instruction on how to prepare the course for the hundreds of students already waiting outside the classroom…

[NB: as afar as I can tell from searching forums and documentation, this is actually how Canvas behaves, but I am willing to be shown otherwise.]

Why can IEEE deny the selection of a DL? In part, there are requirements for DLs now:

DL should visit IT Society local chapters. DL program pays for airfare and travel. Local chapter pays for local expenses (hotel). If traveling to a different continent, visits to two locations are required. DL lectures should be freely accessible to the public (i.e. no registration fees).

A DL from Iran cannot be reimbursed by IEEE because the IEEE is based in the US and has to abide by US law. By the new rules then, scholars from Iran are automatically disqualified from the DL program.

Being a DL is an important recognition: it is arguably an award. It certainly bestows a certain level of prestige. Acceding to this intervention by IEEE sends the message that “if you are from Iran, you can’t get an award.” Once we go down this road we might as well ban conference submissions, membership, and participation in the academic community for scholars from Iran. Why not go whole hog and become a tool of the US State Department? It’s ludicrous.

ITSOC should not sit by and passively accept this “veto” from IEEE: it’s an assault on academic freedom that devalues scholarship on purely political grounds. To not even name the nominee erases the honor to which they are entitled. In fact, they should be given the honor/award with the stipulation that they are exempt from the reimbursement. It is possible to take a stand without violating the law: recognize this scholar and take a public stand against the encroachment of American foreign policy onto an international academic community.

Since there is yet another tier of TPC above the ACs (the Senior ACs), how should one approach the meta-review? One view is that the meta-review is AC’s decision/opinion informed by the reviews, the response, the discussion, and their own reading of the paper. This makes the AC a bit like an associate editor at a journal. This also gives the AC quite a bit of flexibility: if the discussion is limited or not particularly useful, the AC can fill in the gap by adding their own voice. The downside is that ACs might bring more of their own preferences (or biases) to the process.

A different approach is to make the meta-review akin to a panel summary as part of an NSF review. In the panels I have been on, there are N people who write reviews of each proposal, one of whom leads the discussion. There is also a scribe for the discussion who has not written a review: a dispassionate observer. The whole panel (even those who didn’t read the proposal) participates in the discussion. The scribe is supposed to draft a summary/synthesis of the discussion and runs it past the panel for edits until they reach a consensus. The N reviews are still there though, with their diversity of opinion.

I think I might prefer the second model. The setup is a bit different, since authors get to respond to the reviews. The meta-review is supposed to augment the existing reviews by incorporating the discussion and author response. The AC is supposed to guide the discussion, which is a role shared by the lead discussant and program officer in the NSF model. The only problem is that the amount of discussion on each paper is highly variable. It’s sometimes like pulling teeth to get reviewers to respond/interact. Reviewers, for their part, might be participating in 5 different discussions, so context switching to each paper can be tough. But for papers with some reasonable discussion, the meta-review as panel summary might be a good way to go.

One complaint about panel summaries is that they often feel anodyne. However, I think this might be desirable in a meta-review, since it could lead to fewer angry authors. One aspect of the NSF model which I think could be adopted, regardless of how the AC views their job, is running the meta-review past the reviewers. I did this for ICML and got some edits and feedback from the reviewers that improved the final review.

Differential privacy is a promising approach to privacy-preserving data analysis.  Differential privacy provides strong worst-case guarantees about the harm that a user could suffer from participating in a differentially private data analysis, but is also flexible enough to allow for a wide variety of data analyses to be performed with a high degree of utility.  Having already been the subject of a decade of intense scientific study, it has also now been deployed in products at government agencies such as the U.S. Census Bureau and companies like Apple and Google.

Researchers in differential privacy span many distinct research communities, including algorithms, computer security, cryptography, databases, data mining, machine learning, statistics, programming languages, social sciences, and law.  This workshop will bring researchers from these communities together to discuss recent developments in both the theory and practice of differential privacy.

Specific topics of interest for the workshop include (but are not limited to):

• theory of differential privacy,
• differential privacy and security,
• privacy preserving machine learning,
• differential privacy and statistics,
• differential privacy and data analysis,
• trade-offs between privacy protection and analytic utility,
• differential privacy and surveys,
• programming languages for differential privacy,
• relaxations of the differential privacy definition,
• differential privacy vs other privacy notions and methods,
• experimental studies using differential privacy,
• differential privacy implementations,
• differential privacy and policy making,
• applications of differential privacy.

Submissions

The goal of TPDP is to stimulate the discussion on the relevance of differentially private data analyses in practice. For this reason, we seek contributions from different research areas of computer science and statistics.  Authors are invited to submit a short abstract (4 pages maximum) of their work. Submissions will undergo a lightweight review process and will be judged on originality, relevance, interest and clarity. Submission should describe novel work or work that has already appeared elsewhere but that can stimulate the discussion between different communities at the workshop. Accepted abstracts will be presented at the workshop either as a talk or a poster.  The workshop will not have formal proceedings and is not intended to preclude later publication at another venue. Selected papers from the workshop will be invited to submit a full version of their work for publication in a special issue of the Journal of Privacy and Confidentiality.

Submission website: https://easychair.org/conferences/?conf=tpdp2019

Important Dates

Submission: June 21 (anywhere on earth)

Notification: August 9

Workshop: 11/11

Program Committee

• Michael Hay (co-chair), Colgate University
• Aleksandar Nikolov (co-chair), University of Toronto
• Aws Albarghouthi, University of Wisconsin–Madison
• Borja Balle, Amazon
• Mark Bun, Boston University
• Graham Cormode, University of Warwick
• Rachel Cummings, Georgia Tech University
• Xi He, University of Waterloo
• Gautam Kamath, University of Waterloo
• Ilya Mironov, Google Research – Brain
• Uri Stemmer, Ben-Gurion University
• Danfeng Zhang, Penn State University

For more information, visit the workshop website at https://tpdp.cse.buffalo.edu/2019/.

As the US Liaison Chair of IEEE SPAWC 2019, I have received NSF funds to support travel of undergraduate and/or graduate students to Cannes, France for IEEE SPAWC 2019. Having a paper at the workshop is not a prerequisite for these grants and a number of grants are reserved for underrepresented minority students whose careers might benefit from these travel grants. Please share this with any interested students and, if you know one, please encourage her/him to consider applying for these grants.

Absent a compelling reason not to submit code, I think that ensuring some form of reproducibility is important for both transparency and the open communication of ideas. Reviewers already approach reading a paper with some skepticism — the burden of proof is on the authors to make a compelling argument in their paper. But if the argument is largely empirical (e.g. “this heuristic works very well for problem A”) then the burden of proof consists of making a case that the experiments, as described in the paper, were in fact carried out and not mere fabrications. How better to do that than to provide the implementation of the method?

Providing implementations is not always possible: examples abound in multiple fields, including electrical engineering. In antenna design the schematic might be provided in the paper, but the actual fabricated antenna and anechoic chamber are not available to the reviewers. Nobody seems to think this is a problem: reviewers somehow trust that the authors are not making things up. Shouldn’t we trust ML authors as well?

One factor that makes a difference is that conferences are just not as competitive outside of computer science. Conferences have a short review period in which to evaluate a large volume of papers. The prestige conferred by getting a paper accepted to a top CS conference is often compared to getting a paper accepted to a top journal. Authors benefit a lot from the research community accepting their paper. It is only appropriate that they also share a lot.

Let’s take an example. Suppose you are working in academia and developed a new method for solving Problem X. You are going to launch a startup based on this method. How much more appealing would it be to funders if you had one (or more!) ICML papers about how you’ve totally nailed Problem X, showing that you are a total rockstar in the ML/AI community? But your competitive advantage might be at risk if reviewers (and then later the community) has access to your code. So then you write a paper where you discuss the main ideas behind your approach and give the experimental results but no implementation with the 5 other things you had to do to make the thing actually work. In this case you’re getting the stamp of approval while not sharing with the rest of the research community.

Of course, one can imagine that submissions from industry authors might rely on proprietary code bases which they cannot (for policy reasons) provide. An academic conference is about the open and free exchange of ideas, knowledge, and techniques. It seems that a trade show would be a more appropriate venue for showing the results without sharing the methods. I’m not trying to suggest that industry researchers are nefarious in some way, but it’s important to think about the incentives and benefits. The rules for submission (in this case code submission) articulate some of the values of the research community. Encouraging (but not requiring) code submission requires authors to signal (and allows reviewers to consider) whether they agree to the social contract.