Github on and some change

LLM benchmarks like SWE-bench are not trustworthy

Wed, 08 Jan 2025 17:20:47 -0500

If you believe OpenAI’s marketing, their LLM products are automating an increasingly large fraction of software engineering jobs. They substantiate this, in part, by citing how their products perform against various LLM benchmarks. Here’s an example where OpenAI touts GPT-4o’s performance on a benchmark called SWE-bench:

Each sample in the SWE-bench test set is created from a resolved GitHub issue in one of 12 open-source Python repositories on GitHub. Each sample has an associated pull request (PR), which includes both the solution code and unit tests to verify code correctness.
[…]
We found that GPT-4o’s performance on the best-performing scaffold reaches 33.2% on SWE-bench Verified, more than doubling its score of 16% on the original SWE-bench.

These benchmarks differ a bit, but most of them are structurally equivalent:

They have a ~static list of problem statements. Sometimes these are coding problems, sometimes they’re math problems.
They feed each problem statement to the LLM, and take the response.
They take each response, and compare it to a static list of known correct answers for each problem.
Finally, they calculate what percentage of questions the LLM got right.

Some benchmarks get a little fancy - FrontierMath rates how “hard” the math problems are - but they’re all roughly shaped like this. Some benchmarks, like SWE-bench, publish all the problem statements publicly. Some, like, FrontierMath, don’t, but critically, have to send the problems to OpenAI to evaluate its models.

Now imagine for a second that you’re an LLM company, trying to make your model better at coding or math. Obviously you’re going to scrape Github and include its code, issues, and PRs in your training set. And obviously you’re going to look at math problems your users are solving, and use that in future training runs. The upshot of this is that the vast majority of public and non-public benchmarks’ question sets are in your training set.

When a benchmark has been undermined in this way, you see the LLM perform extremely well in-sample (on questions it’s been trained on), but its performance dramatically decreases out-of-sample (like, when it actually runs in the real world). That’s why avoiding this is, like, ML 101; training on your test set means benchmarks no longer give you an accurate picture of how good your model is.

And this is the shape of what’s happened with SWE-bench; some folks at York University discovered that ~94% of SWE-bench’s problems came from data that existed during the training of e.g. OpenAI’s GPT-4o. When they restricted the dataset, GPT-4o’s performance dropped by ~90%, to a success rate of 3.8%:

In addition, over 94% of the issues were created before LLM’s knowl-edge cutoff dates, posing potential data leakage issues.
[…]
After carefully analyzing the passed instances from the SWE-Agent + GPT-4 model with the new dataset, SWE-Bench+, we observed a decline in the pass rate, dropping from 3.97% (as seen on the refined SWE-Bench) to a resolution rate of 0.55%. We further evaluated SWE-RAG + GPT-4, SWE-RAG + GPT-3.5, and AutoCodeRover + GPT-4o models on the new dataset to verify our findings, where the resolution rates of the models drop significantly, which are 0.73%, 0.55%, and 3.83%, respectively.

With FrontierMath, OpenAI’s o1 scored less than 2% when the benchmark was released in November 2024. At the end of December, OpenAI announced that o3 solved 25% of the problems. (Interestingly, the public hasn’t been given access to the model.)

Update, 2025-01-18: An Epoch AI employee confirmed that OpenAI did have access to the questions and solutions for the FrontierMath benchmark. Worse, they were contractually-forbidden from disclosing this, so the confusion about whether FrontierMath was a trustworthy benchmark was almost certainly intentional on OpenAI’s part.

To be clear, I’m not claiming that OpenAI is knowingly cheating on benchmarks. I don’t think they have to be, in order for this to happen. They’re a private company with a closed model, and internally, they’re hoovering up as much data as they can possibly get. This leakage is just what I’d expect to happen by default.

But I do think that you absolutely have to view LLM benchmarks from a position of default-distrust, especially given the amounts of money sloshing around in the AI industry. That’s what’s required for honest science. The real question, in my mind, is whether or when we’ll see autonomous software engineering, or math research, being done anywhere.

Thanks to Guanqing Ou for feedback on this post.

Predictions for 2025

Sun, 05 Jan 2025 19:28:59 -0500

Following up on my predictions for 2024, here’s a bunch of predictions for this year. As before, all dollar values inflation-adjusted for start of 2025.

US

National

❌ US enters a recession: 10%
❌ Median YoY CPI inflation <2%: 40%
✅ US GDP >$30T: 60%

NYC

Housing

❌ 2024 estimated annual population change is <=0%: 80%
❌ 2024 new housing starts down from 2023 (27,980): 70%
❌ 2025 new housing permits issued up from 2024: 50%
❌ Median listing $/sqft in the NY/Newark/Jersey City MSA <$500: 20%

Congestion pricing

✅ Remains in place, without significant rollbacks, by the end of the year: 70%
✅ I notice reduced traffic volumes in my neighborhood: 10%

Second Avenue Subway

✅ Phase 2 construction starts before end of year: 90%
❌ Phase 2 budget exceeds $9 billion at some point: 20%
❌ Future phases officially cancelled: 10%

California

Housing

✅ 2024 estimated population change is >=0%: 70%
2025 new housing permits issued up from 2024: 50%
✅ Median listing $/sqft in the SF/Oakland/Hayward MSA <$640: 20%

HSR

I’ll assess these only if the CAHSR releases a new project update report, with new figures. (I think the next one is scheduled for early this year.) If they don’t by the end of the year, I’ll annul these predictions.

(Annulled) P50 estimate of Bakersfield-Merced opening delayed to at least 2034: 60%
- I consider the “P50 probability envelope” the official estimate, i.e. currently the official estimate is “end of 2033”.
- The Authority changed how it does its reporting - it no longer reports percentiles for timelines - so I’m annulling this prediction. I think the change is a negative sign about the timeline, though.
✅ Base YOE cost of Bakersfield-Merced rises to or above $33B: 70%
✅ Project officially cancelled: 20%
- I’d consider any change to the project that abandons its current goal of serving SF-LA in <3hrs as cancelling it.
- The plan abandons the idea of connecting directly to LA, instead ending at Palmdale, ~80 miles away.

Rest of world

Canada

✅ We decide this year to move to Canada: 25%
✅ Trudeau resigns PM before the election: 40%
❌ A motion of no confidence in Trudeau is introduced: 50%
- ❌ … and he survives it: 20%
❌ Conservatives win a plurality of the legislature: 60%

Ukraine

❌ Admitted to NATO: 20%
❌ Cedes territory to Russia: 30%

China

❌ Enters a recognized recession: 10%
❌ Formally revokes Hong Kong SAR status: 10%
✅ Xi Jinping still in power at end of year: 90%
❌ Invades Taiwan: 10%
❌ BRICS loses a member: 10%
- Any of Brazil, Russia, India, China, South Africa, Egypt, Ethiopia, Iran, and the UAE, or any member that joins during 2025.

Argentina

✅ YoY inflation hits <100% at any point during the year: 50%
✅ La Libertad Avanza wins a plurality of the seats up for election in the Chamber of Deputies: 70%

Tech

AI

✅ OpenAI announces GPT-5: 70%
✅ … and it’s available to consumers: 60%
❌ OpenAI gets a new CEO: 30%
❌ We achieve AGI: 5%

Other

❌ Alphabet, Amazon, or Apple are forced via lawsuit to spin off at least one company: 10%
❌ At least one company offers widely-available self-driving cars: 30%
❌ Twitter files for bankruptcy: 20%
❌ Amazon, Google, Apple, or Steam accept Bitcoin: 5%
✅AGDQ 2025 raises at least $2.5M: 60% ($2.6M)
❌ Apple releases a foldable iPhone: 10%

Pop culture

❌ Marvel films will have total revenue over 1.5 billion dollars (international box office): 60%
❌ Marvel films will average greater than 650 million in international box office: 40%
✅ There are >5 movies with domestic box office >250 million: 25%
❌ There are >10 movies with domestic box office >250 million: 10%
❌ An A24 movie will win the Best Picture Oscar: 40%
✅ Kieran Culkin will win the Best Supporting Actor Oscar: 75%
❌ Challengers will win an Oscar: 60%
❌ The live action Snow White will make more than 200 million dollars in domestic box office: 60%
❌ A streaming service will close down / be merged into another: 30%
- Max, Peacock, Paramount, Hulu, Disney+, Amazon Prime Video, Netflix, Apple TV+, Mubi
✅ The book that spends the longest at the top of the NYT bestseller list is a romance: 60%
✅ The highest grossing movie of the year domestically is a sequel or part of an ongoing franchise (like Marvel, with previous entries): 80%

Sports

✅ The NFC champion will win the Super Bowl: 40%
✅ The Mets will make the playoffs: 50%
✅ The Knicks will have a winning season: 70%
❌ The Eastern Conference champion will win the NBA finals: 50%

Following up on my 2024 predictions

Tue, 31 Dec 2024 14:14:53 -0500

Here’s how I did with my 2024 predictions.

Last year’s predictions

US

Congestion pricing

Begins collection by EOY: 70% (did not happen)
Collects the expected $1b annualized (factoring in discounts): 40% (did not happen)
I notice reduced traffic volumes in my neighborhood: 30% (did not happen)

The first one was a near miss; instead, collection is scheduled to start on January 5th, 2025, after the governor (in my opinion) illegally paused the program until after the federal elections, allegedly to help Democrats in vulnerable New York districts.

Obviously the revenues and traffic impacts didn’t happen, either. The top-line tolls were also cut from $15 to $9; I expect annual revenues to be correspondingly lower, and the impact on congestion to also be pretty minimal.

Second Avenue Subway

Phase 2 construction starts before end of year: 50% (did not happen)
Phase 2 budget exceeds $7 billion at some point: 80% (happened, but disqualified)
Future phases officially cancelled: 10% (did not happen)

I think the first bullet would’ve happened, but because of congestion tolling delays, it was delayed. The first contracts for doing construction for Phase 2 are just now being awarded, and I expect them to start breaking ground early next year.

The second bullet shouldn’t count as a success, IMO; by October last year the official estimates for Phase 2 already exceeded $7b. My numbers were just outdated.

California HSR

P50 estimate of Bakersfield-Merced opening delayed to at least 2034: 60% (did not happen)
Base YOE cost of Bakersfield-Merced rises to or above $33B: 50% (did not happen)
Project officially cancelled: 30% (did not happen)

I whiffed on all three of these, mostly because of an oversight on my part - the California High Speed Rail Authority only issues project update reports (where the project timeline & budget estimates live) once every two years, and this year they didn’t issue one. As a result, the estimates are all officially unchanged, so I’ll call this a loss.

Unofficially, though, the fact that the Authority is recently targeting $35B in revenues and shopping around numbers like $130B for the whole project indicates to me that the project timelines will be delayed, and budgets will be increased, next time around. I was wrong about the project management bits, but I think I was right about the fundamentals of how the project is going.

Other

US enters a recession: 20% (did not happen)
Donald Trump
- Goes to trial before the general election in any of his current cases: 10% (happened)
- Wins the Republican primary: 90% (happened)
- Wins the general election: 40% (happened)

Very happy about the first, and extremely unhappy about the rest.

Rest of world

We decide this year to move to Canada: 50% (did not happen)

We put the decision off a year, since we can afford to do that.

Ukraine
- Admitted to NATO: 20% (did not happen)
- Cedes territory to Russia: 20% (did not happen)
Labour wins UK elections: 90% (happened)
China
- Enters a recognized recession: 40% (did not happen)
- Formally revokes Hong Kong SAR status: 10% (did not happen)
- Xi Jinping leaves power: 10% (did not happen)
- Invades Taiwan: 10% (did not happen)
BRICS loses a member: 70% (happened)
- Any of Brazil, Russia, India, China, South Africa, Egypt, Ethiopia, Iran, Saudi Arabia, and the UAE, or any member that joins during 2024.

Both Saudi Arabia and Argentina, who were going to join BRICS, did not end up as members at the end of the year. I’m going to call this a win in letter only, because I listed SA explicitly. However, I’m encouraged by just how brittle BRICS has been overall.

In general I’m pretty optimistic about the direction things are going in the world. The one big exception here is the war in the Ukraine, which I’ve been following closely, and am very sorry to see things trending in a bad way.

Tech

OpenAI
- Announces GPT-5: 90% (did not happen)
- Releases GPT-5 for consumer use: 80% (did not happen)
- Gets a new CEO: 50% (did not happen)
We achieve AGI: 5% (did not happen)

I was surprised a lot by this. Feels to me like OpenAI is trudging towards failure, which I pessimistically did not expect to happen.

Alphabet, Amazon, or Apple are forced via lawsuit to spin off at least one company: 25% (did not happen)

This might still happen in 2025! The DoJ formally sought to force Google to spin off Chrome in November; we’ll see what the result is.

At least one company offers widely-available self-driving cars: 30% (did not happen)

Waymo expanded operations but is not what I’d consider widely-available.

Tesla stops selling a product named “Full Self-Driving”: 40% (happened)

They renamed FSD to “FSD (supervised)”, which I think qualifies.

Twitter files for bankruptcy: 70% (did not happen)
Amazon, Google, Apple, or Steam accept Bitcoin: 5% (did not happen)

How’d I do?

Here’s a calibration chart of my predictions, bucketed into three groups - things I thought were 0-33%, 34-66%, and 67-100% likely to happen. The red line is how often things in each bucket should’ve happened if I was exactly correct about my probability estimates; the blue line is how often they actually happened. The ideal is for the two lines to be exactly the same.

Not amazing! A few things went wrong. First, I was waaay overconfident across the board - stuff I thought would happen ~17% of the time actually happened 0% of the time, and stuff that I thought would happen 46% of the time only happened 25% of the time. Second, I was particularly inaccurate for things I thought were sure to happen - for stuff in the 66-100% category, only ~43% came true.

This happened for a few reasons. Three of my whiffs could never have happened - the CA HSR project update snafu that I could’ve avoided if I’d doine a little more research. And a lot of stuff was conditional on congestion tolling happening in NYC, so when Hochul stopped that from happening, several of my predictions immediately became wrong.

Overall, I think I need to be a lot less certain in general. I think that’s why this exercise is valuable - it feels good to know how wildly off my beliefs are, and that I should be particularly careful for things I think are obvious.

Notes from Midtown East Casino Proposal Town Hall, 2024/01/11

Thu, 11 Jan 2024 18:44:55 -0500

Below are some notes I took during a town hall organized by Kristen Gonzalez, on the topic of a proposal for a casino in Midtown East (38th to 41st St, between First Ave and FDR). For some background, see coverage in the Gothamist.

The meeting was structured as follows. First, a member of the gaming location board spoke to give background on the proposal & process. Then, some of the six voting members of the Community Advisory Committee each spoke for a short amount of time, followed by representatives of some locally-powerful groups. Finally, representatives from the proposing group spoke and held a Q&A moderated by Kristen Gonzalez.

I didn’t take notes for the first two speakers, but one was a member of the gaming location board (Jerry) & described process. The next speaker was pretty emphatically a no. Kristen Gonazalez spoke briefly between each speaker, primarily to calm nerves, remind folks that reps should be appreciated for taking time to listen & thanked.

I was pretty impressed with Kristen Gonzalez; despite an extremely raucous crowd that heckled the speakers repeatedly, she ran a smooth event, calming nerves between speakers, organizing it in a sensible fashion that meaningfully informed the community both about the facts and each speaker’s likely position on the issue. The one exception was the Q&A section, where in an attempt to group up attendee questions, she ended up asking vague questions like “Can you talk about public safety?” which had the effect of softballing what should have been much more hard-hitting concrete questions about the proposal.

I was very unimpressed by the attendees from the Manhattan East Community Association (MECA), who seemed primarily there to advertise for their organization. They spent basically all their time talking about unrelated events they’d run, in contrast to the other speakers who stuck pretty closely to the topic at hand. Maybe they didn’t get the memo?

I expect Robert Gottheim (obo Jerry Nadler) to get the most coverage. He had by far the most memorable moment of the night, where in response to a heckler yelling “This is blood money”, he responded “It may be blood money, but the money is coming.” It was a pretty shocking thing to say in the moment, but it fit with his overall (IMO reasonable) message that residents should prepare themselves for the likely outcome that Manhattan will get a casino somewhere in the near future.

Ultimately, it was a lot of fun to attend! The atmosphere was pretty free-wheeling, with attendees interjecting pretty regularly, and it had the feeling of a small-town meeting. I’d highly recommend events like this if you’re at all interested in local politics or shooting the shit with your neighbors in New York.

Keith Powers, NYC Councilmember, District 4
- I get questions all the time about what’s happening here
- Voices of community & nearby are important
- We have a lot of time (180) days to hear feedback
- Numerous proposals and only a few that will make it through
- I live just a few blocks away, have a personal stake
- Doesn’t state personal up/down lean
Lizette Chaparro for Mark Levine, Manhattan Borough President
- Notes hundreds of attendees, thanks for coming tonight
- A lot of what I wanted to say has already been echoed
- One of five proposals that we’re aware of in Manhattan
- Aware of magnitude of each proposal
- Thinking of impacts on local businesses, traffic, pedestrian flows
- Thinking of benefits: housing, jobs, public amenities
- Long process ahead of us
- Doesn’t state personal up/down
Robert Gottheim for Jerry Nadler, US Rep NY-12
- Nadler doesn’t appoint anyone to the CAG
- Will use bully pulpit to advocate on your behalf
- Issue that people are passionate about, controversial
- 14th - 96th E side, up to 104th on W side
- Five proposals for casinos, Manhattan is going to get one of them, this is where the money is
- Reality is: Manhattan will probably have a casino
- Our job is to decide: where is the best location? This may not be it
- I’m here to hear from you & your concerns
- This is close to major residential (PCV/ST), impact on children, quality of life
- Some heckling
  - “It may be blood money, but the money is coming. Will we want money?”
  - We are getting casinos in New York. I wish we weren’t. They prey on the most vulnerable, it’s not what we should be doing. But we have to deal with reality.
  - Please speak up, we want to hear from you
Susan Steinberg, STPCV Tenants Association President
- Development plans are extremely ambitious
- Some might really appreciate this eyesore being put to use
- Need to consider effects on (traffic, congestion, …)
- Science Park & Research Center, 1st Ave & 25th, challenges, construction disruption for years
- Is another megaproject good for our community? (waits for people to yell NO)
- Pay close attention to developers’ description of plans
- You and me will be impacted for years to come by this project. You have to convince us that this is a benefit. Otherwise, no thank you
Michael-Ann Rowe, VP Manhattan East Community Association
- Implant from Canada, live in Murray Hill
- Describes MECA
- Story about making a different project halt due to environmental concerns
- Introduces Kevin O’Keefe, MECA president and CB-6 member
Kevin O’Keefe, President MECA
- Story of family being born here
- We’ve heard from many in favor, against, and undecided
- CB-6’s vote earlier in the year was too early
- Want to push for more affordable units

Kirsten Gonzalez describes comunity boards

Sandy McKey, CB-6 chair
- 11 days into tenure
- Describes an earlier Jan 2023 CB-6 resolution in favor of the project
- Describes Mar 2023 CB-6 resolution
  - Noted a prime site for affordable housing
- Some heckling, pauses and waits
- CB-6 applications are open
Janni, Freedom Plaza project team
- Working with Soloviev group for ~decade, Mohegan group for ~1yr
- Mohegan Tribe driven by respect for the Earth, have been a part of the northeast for many years
- 1325 housing units, 513 affordable
- Museum of Freedom & Democracy, community food market, retail, daycare
- Thousands of union jobs
- 0 MIH units have been built
- We’re making this affordable housing possible
- Community investment fund: 2% of profits annually, minimum $5M
  - Nonprofit not controlled by Soloviev or Mohegan
- Freedom Plaza, big park with events, 100% ADA accessible
- Contact info on bottom of last slide: [email protected]">[email protected]
- Introduces executives from various project team groups
- Literally all of them are older white men
Q&A
- Why was the site unused all these years?
  - They sold part of it?
  - We had a plan to build high-rises, but we decided to change the plan
- What does affordable housing mean?
  - How much?
  - Why is the housing connected to the casino?
  - Will any of the housing be used to address the migrant crisis?
  - Discusses definition of affordable housing
  - Want this to be a “live to work” community
  - Gestures at incentives re: housing attached to casino
  - Without the casino, this housing would not be built
- Transit
  - We’re trying to maximize the amount of public transit that’s utilized
  - 75% will be using personal vehicles
- Public safety?
  - Safe & clean environment for our guests
  - Work with gaming commission, police, in-house police
  - Will be one of the safest blocks in NYC
- Public safety & the UN?
- Parking
  - There are several parking garages in the neighborhood (heckling re: they’re all full)
  - Our peak hours don’t overlap with peak traffic hours
- Mental health
  - We’re going to have a physical presence in the casino
  - We’re not required to do this but we think it’s important
  - Identify problematic individuals and take action
- CB6 questions
  - CB-6 is advisory, we don’t have a binding role, it’s our advice to our elected officials
  - “You’d better listen” heckling
- Traffic
- Community fund: how will it be used? Is it a binding commitment
  - Yes
  - We’ll ask community leaders to form a 501c
  - Board made of community leaders from all the different community organizations
- Flooding engineering
  - Our environmental engineer isn’t here
- Other sites
  - Hudson Yards
  - Broadway
  - Lots of heckling
  - Bronx
  - Yonkers raceway
- Does Manhattan have to be one of the casinos?
  - Not legally
  - Lots of cheering
  - Gonzalez clarifies it’s likely
  - Gaming board doesn’t have a mandate to locate in any specific area
- Timeline
  - Nothing is going to happen overnight
Public comment
- I only listened to the first few minutes of this, but what I heard felt pretty unproductive to me

Predictions for 2024

Sun, 31 Dec 2023 19:06:51 -0500

Here’s a bunch of predictions for next year, along with rough probabilities. All dollar values inflation-adjusted for start of 2024.

US

Congestion pricing
- Begins collection by EOY: 70%
- Collects the expected $1b annualized (factoring in discounts): 40%
- I notice reduced traffic volumes in my neighborhood: 30%
Second Avenue Subway
- Phase 2 construction starts before end of year: 50%
- Phase 2 budget exceeds $7 billion at some point: 80%
- Future phases officially cancelled: 10%
California HSR
- P50 estimate of Bakersfield-Merced opening delayed to at least 2034: 60%
  - I consider the “P50 probability envelope” the official estimate, i.e. currently the official estimate is “end of 2033”.
- Base YOE cost of Bakersfield-Merced rises to or above $33B: 50%
- Project officially cancelled: 30%
  - I’d consider any change to the project that abandons its current goal of serving SF-LA in <3hrs as cancelling it.
US enters a recession: 20%
Donald Trump
- Goes to trial before the general election in any of his current cases: 10%
- Wins the Republican primary: 90%
- Wins the general election: 40%

Rest of world

We decide this year to move to Canada: 50%
Ukraine
- Admitted to NATO: 20%
- Cedes territory to Russia: 20%
Labour wins UK elections: 90%
China
- Enters a recognized recession: 40%
- Formally revokes Hong Kong SAR status: 10%
- Xi Jinping leaves power: 10%
- Invades Taiwan: 10%
BRICS loses a member: 70%
- Any of Brazil, Russia, India, China, South Africa, Egypt, Ethiopia, Iran, Saudi Arabia, and the UAE, or any member that joins during 2024.

Tech

OpenAI
- Announces GPT-5: 90%
- Releases GPT-5 for consumer use: 80%
- Gets a new CEO: 50%
We achieve AGI: 5%
Alphabet, Amazon, or Apple are forced via lawsuit to spin off at least one company: 25%
At least one company offers widely-available self-driving cars: 30%
Tesla stops selling a product named “Full Self-Driving”: 40%
Twitter files for bankruptcy: 70%
Amazon, Google, Apple, or Steam accept Bitcoin: 5%

The US government is 13x better at healthcare insurance than private companies

Mon, 27 Nov 2023 23:23:57 -0500

Per a KFF report, the US government spent less than 1.3% of Medicare expenses on overhead in 2021 – the remaining 98.7% went directly to paying for healthcare. For comparison, private Medicare (called “Medicare Advantage” by some) spent 17% on overhead, making the government ~13x more efficient at running healthcare insurance than private companies. Even this stacks the deck in favor of private companies; the 1.3% figure includes spending on overseeing private Medicare.

This isn’t new, either – the government has been much, much better than private companies at this for a long while.

Debugging stories: the inconsistent database

Mon, 05 Dec 2022 09:33:10 -0500

Recently at work I ran across a bug which I thought was kind of interesting, so I figured I’d write it up.

At my company, my team builds and owns the infra that helps other engineers work with schemas - descriptions of how datasets and APIs are structured, that can be shared and reused across different languages and applications to interoperate with the same data and APIs.

One system we rely on is a database that we use to register and store these schemas. (Internally, we call this the schema registry.) A user came in reporting that they’d made a change to their schemas awhile ago, but they weren’t showing up in the registry, and I started digging in.

The first thing I did was actually verify that the new version of the schemas weren’t actually registered. One of these schemas we’ll call “schemaB”:

curl http://schema-registry/subjects/schemaB/versions[]

Yep, it was missing all right. That confirmed that there was a real problem.

The next thing to do was to figure out what happened. The thing that actually stores new schema versions in the registry (the schema publisher) is an application that we own and emits logs, so I went to take a look at those logs. The logs showed something like this happening:

Checking if [email protected] exists...Attempting to register [email protected]...Success!Checking if [email protected] exists...Attempting to register [email protected]...ERROR from schema registry: schemaB's dependency [email protected] does not exist!

Here, “1.2” and “1.1” are new versions of the schemas that the user is trying to register. In this case, schemaB depends on schemaA, so we try to publish [email protected]">[email protected] first, and after that succeeds, we try (and fail) to publish [email protected]">[email protected].

Note that this is weird - the schema registry was telling us that it registered [email protected]">[email protected], and then immediately afterwards, complaining that it didn’t exist. Wait, does it exist?

curl http://schema-registry/subjects/schemaA/versions[1.1, 1.2]

Yep, looks like it does. So what I knew was:

The registry sometimes knows that [email protected]">[email protected] exists, but
Sometimes it doesn’t.

At this point, I realized that I was asking a question, and different entities were answering, with different views of the world. That implied that the schema registry was a distributed system.

To confirm this, I re-ran the above request a few times:

curl http://schema-registry/subjects/schemaA/versions[1.1, 1.2]curl http://schema-registry/subjects/schemaA/versions[1.1, 1.2]curl http://schema-registry/subjects/schemaA/versions[1.1, 1.2]curl http://schema-registry/subjects/schemaA/versions[1.1]

Bingo - a smoking gun - one of the schema registry’s request-servers only knew about 1.1, and not 1.2. This is typical of a distributed system that’s eventually-consistent - after you do something that changes state in some way, not all parts of the system get updated immediately, and it might take some time for everyone to get on the same page.

I re-ran the publisher, and:

Checking if [email protected] exists...[email protected] exists, skipping.Checking if [email protected] exists...Attempting to register [email protected]...Success!

I reported back to the user that their problem had been fixed, and discussed some potential fixes with the team.

In our case, we ended up adding some retry logic to the publisher - in the case where a registration attempt failed because of missing upstream dependencies that we’d just published, we’d back off for a little while, then retry publishing (repeating this for a fixed number of attempts in case of failure).

But ideally (at least from the perspective of making the client simpler), the registry should be strongly-consistent instead of just eventually-consistent. With strong consistency, the registry would basically wait until all its nodes were aware of the update before returning a successful result. That would eliminate the intermediate inconsistent state - when the registry returned a successful publishing result, subsequent requests depending on that schema would be able to access it. That’d prevent errors like the one we ran into here, at the cost of making registration requests blocking operations.

Setting this blog up on Github Pages

Thu, 03 Nov 2022 22:30:37 -0400

As a birthday present to myself this year, I finally decided to bite the bullet and quit my Twitter addiction. I signed up for Mastodon, which seems nice so far (you can find me here). I’m also hoping to do a little bit more writing with all the spare time I now have on my hands. Some of that will be on Mastodon, but hopefully the longer-form bits will start to appear here.

My blog’s been limping along for some time - the whole infrastructure of writing a post, publishing it, and actually serving the site was basically hand-rolled a long time ago by yours truly. As might be expected, this setup, held together by paperclips and duct tape, fell over all the time. To make the process of writing easier on myself, I decided to migrate to Github Pages. The whole process was surprisingly straightforward, so I thought I’d write it up in case others would find this useful.

What’s Github Pages?

Github is a website that people use to track and share the source code for software they write. One of the satellite services that Github also runs is called Pages. People on Github can host a small personal website, or small project-based websites on Github Pages. I use Github to store the source code of this blog (all the post text and images, theme, etc), so Pages seemed like a natural fit to actually host the site as well.

How do things work now?

Before I dig into the guts of how I set this up, it’s worth talking through what the end result looks like, from writing to publishing.

First, to write a new post, I create a new Markdown file in the repository for this blog. Each post is a separate file.

Once it’s ready, I create a commit on the blog repository’s main branch and push it to Github:

# Add the new post to Gitgit add content/posts/my-new-awesome-post.md# Create a new commitgit commit -m "Add new post: setting-this-blog-up-on-github-pages"# Push it to Githubgit push

If I’m feeling lazy, sometimes I’ll do these steps in VSCode instead, which has a nice GUI for git, but feels slower. I’m not sure if it’s actually slower than typing everything out, though!

Pushing that commit to Github fires off some stuff on Github which regenerates the site in about a minute:

And that’s it! Pretty simple if I do say so myself.

How’s this set up?

For the most part, I just followed Github’s excellent instructions, which you can find here.. You should probably just read their docs, but here’s what I did specifically:

As the zeroth step, I had previously set my website up using Hugo, a blogging software.

First, I went to my repository settings page on Github, navigated to the “Pages” tab, and selected “Github Actions” under “Source”:

This tells Github that you want your site’s content to be built and deployed via a Github Action (more on that in a bit) and not a custom deployment on a dedicated branch.

The next step is to point your domain to Github. I went to my domain registrar’s DNS configuration page. I had registered the domain under gandi.net, so I signed into their web console, navigated to my domain’s DNS records, and created the following table:

A couple things are worth explaining here:

The first entry is autogenerated by my domain registrar; I wish I could delete it, but they won’t let me.
Lines 2-5 are necessary; they point the top-level domain (seinmastudios.com) to Github’s servers.
Line 6 sets up an alias to point www.seinmastudios.com to seinmastudios.com instead. This is useful, because Github will read this configuration later and set up some nice redirection magic.

I saved these records, and then went back to Github to test my DNS configuration. On the same page as before, under “Custom domain”, I typed in my domain and hit save, which triggered Github to start running a test:

This was by far the trickiest part of the whole setup, because I had to futz with my DNS records a couple times to find the right settings, and DNS records end up being cached unexpectedly. Basically what I ended up doing was walking away for five minutes (the TTL I set in the DNS records above) between changes.

Anyways, once that’s done, it’s time to set up a Github Action to deploy the site to your Github Pages website. To do this, I clicked the “Actions” tab of my repository, clicked “New workflow”, and picked the Hugo workflow (which is the blog website software I use):

This action comes with a configuration file. I didn’t have to modify it at all - just clicked “Start commit” and committed it directly to my main branch.

At that point, the Action kicked off on my latest master commit, built my website, and pushed my changes - I went to my website and it was working!

Random musings

I’m continually impressed at how easy and straightforward it is to set up a Github Action nowadays that does something that would’ve taken days or weeks of effort on my part. Someone at Github definitely deserves a raise for good work done here. The little bit of configuration in this post allowed me to delete 255 lines of gnarly Ansible code. I’ll miss futzing with it a little, but I think overall I’m much happier - I’ve learned the hard way that less code is almost always better!

The one thing I’ve encountered before with Github Actions is that you have to be careful about depending on third-party actions not maintained by either Github or a company directly invested in maintaining it. For instance, a lot of my side projects used to rely on a Github Action to build Docker container images, which was authored by a (well-meaning) individual not associated with Github, Docker, or any other long-lived entity. This was great for a few months, but predictably the thing fell out of date with Github and Docker APIs, which meant it stopped working all of a sudden one day. I had to find a replacement and rewrite a bunch of YAML configuration for the new action, which was a pain in the neck.

I’m not too worried about the setup I’ve done here, though - this is all using native Github-maintained stuff, so I look forward to a long shelf life for the current arrangement.

Catching Us Up to 2018

Tue, 02 Jan 2018 12:03:41 -0800

Well, it’s been a year since my last blogpost, which caught us up to 2017. Let’s bring us up to speed again, shall we? Here’s the month-by-month highlights once again:

Jan 2017: Visited my father in Eugene, where we went hiking at Spencer Butte.

Feb 2017: Pretty quiet month! Mostly just classes at SF State.

Mar 2017: Went to Cal Academy and the Musée Mécanique, which is an arcade filled with old-school arcade games. Really cool stuff.

Apr 2017: We saw Hamilton at the theatre, which was AMAZING. Went to see a Giants game at AT&T park, where we got helmet nachos. My car got scratched up by someone, and worked through the repair and paperwork process with them. Also, went to a friend’s wedding in the South Bay, and had a blast.

May 2017: I received a sous vide kit as a gift, and my life was never again the same. My mother and sister came to visit at the end of the month, for my graduation! Also, I graduated from SF State!!! Guanqing and I went to Las Vegas for a weekend to celebrate. We went to all the buffets, saw the Chippendales, and got room service for the first time in my life. It was everything I dreamed of, and more.

June 2017: I started work at Grand Rounds, a healthcare startup. Went back to Minneapolis for my sister’s high school graduation. Went to another friend’s wedding in the South Bay. Started apartment-hunting for a pet-friendly place. Walked in the Pride Parade downtown SF.

July 2017: This was a huge month. Visited Chicago over Independence Day weekend with Guanqing. We visited the University of Chicago, went on an architecture tour on the river, and visited museum campus. Also hung out with Maggie at Anime Midwest, which was a lot of fun. We moved into our new place in South San Francisco – big ol’ house in a nice neighborhood, which we were super fortunate to find. WE ALSO ADOPTED A CAT, Max, who has touched our lives immeasurably. Rose came to visit at the end of the month from Japan – really missed her, showed her some of the touristy places in SF.

August 2017: Went to see Harry Potter in concert. Threw a housewarming party for our Bay Area friends – note to self, next time don’t cook all the food during the party, this is a terrible idea. Also went to see the cast of the West Wing Weekly come do a live recording in SF with Aaron Sorkin. Guanqing went to China with her parents, which was disorienting for a week or so. Sort of weird to be on your own again after a year of that not being the case.

September 2017: Took the GRE general exam and did exactly as well as I did on practice exams (heck yes). Went to Lovejoy’s Cafe with Guanqing. I love that place.

October 2017: Went to Github Universe, a tech conference, with two coworkers; later, went to GraphQL Summit with a few other coworkers. Also, Guanqing and I got engaged!!! Probably should have led with that. My mother came to visit us, and we went to take a look at wedding ceremony sites in Golden Gate Park’s Botanical Gardens. Went to see Atul Gawande in SF. Signed up for a South SF library card! Started on the wedding dress hunt.

November 2017: Went to Lovejoy’s Cafe for my birthday! Ugh that place is so, so good. Went hiking in the Zwierlein Picnic area in the South Bay, and walked around the Japanese Friendship Garden down by San Jose. My mother and sister came down for Thanksgiving, so we went to see the Christmas tree lighting downtown.

December 2017: Went to see Mike Duncan, author of the History of Rome podcast, speak at the Wharf in SF. Guanqing’s chamber orchestra performed Messyah at a few different places around the Bay. We tricked two of our friends into cat-sitting for us over the holiday break. Then we jetted off to New Hampshire to meet Guanqing’s parents, and to Minneapolis to talk with my mother and sister. Got everyone on the same page wrt the wedding, and explored Olin College, Guanqing’s alma mater. I really missed the snow. I missed it so much I shoveled a driveway for fun. Took my mother and sister out to see Phantom of the Opera at the Orpheum, which was stellar.

Catching Us Up to 2017

Thu, 12 Jan 2017 12:58:32 -0800

It’s been a long time since my last real update in July 2014. What’s happened since then? Here’s the highlights, month by month.

August 2014: Visited Rose Turck in Springfield IL, had a blast hanging out
September 2014: Visited my dad in Eugene OR and helped him move into his new place for the first two weeks or so. We went crabbing, which was really fun and ended up being a terrible way to feed ourselves but a great bonding experience. I also started applying to jobs towards the end of the month.
October 2014: I interviewed in-person at Clever in SF, did OK on the whiteboard portion, utterly bombed on the coding portion (which is the opposite of what I expected).
November 2014: I interviewed at a half-dozen other places. Things started feeling real, and I started to get excited about actually moving out to California and doing a thing outside of school.
December 2014: I interviewed in-person at Curbside, and after they clarified my school status, they made an offer that I accepted! This was a hugely emotional moment for me, I can’t understate how good it was for me to know that there was a light at the end of the tunnel.
January 2015: This month is basically a blur to me, I was so ridiculously busy. I booked an Airbnb in Mountain View and flew out pretty much immediately, and when I wasn’t apartment-hunting constantly, I was working. My hosts were amazingly helpful, as was Drew (our company HR guy) during the whole onboarding process.
February 2015: I moved into permanent lodgings at the start of the month and slowly started to accumulate actual furniture and home-y things. At Curbside, I’d just gotten the hang of things and started work with Stephen, a senior engineer, on overhauling the retailer data pipeline.
March-June 2015: ??? Lots of working, I guess. Also, lots of going to up to SF.
July 2015: Went to Anime Expo with a bunch of SATers. In retrospect, it was the last time I’d see any of them (for a long time, anyways). AX was enormous and I’m really glad I went. I’m also really glad we had a couple of locals in our group who knew great places to get food and could drive us there! Also, Rose comes back to visit in SF with me for a few days!
August 2015: Went back to Iowa to see a friend on their farm, and visited the Iowa State Fair. Rekindled my love of animals, ugh it was so much fun.
September 2015: I started classes at Foothill College in preparation for going back to finish my degree. Also went to my first Big Book Sale held by the SF public library, oh my god I can’t believe I’d been missing those my entire life.
October 2015: Went hiking a lot. Visited my dad in Eugene.
November 2015: Wine tour in Napa. Curbside office remodel (mostly packing everyone a little closer together). Bay Area Science Festival.
December 2015: Visited family in MN for the holidays. At the end of the month, moved into an apartment in SF with my girlfriend (this was a harrowing process that I will never quite recover from).
January 2016: Began the commute from SF to Palo Alto, which is two hours one-way by public transit. Stephen moved on from Curbside, so I spent a bunch of time learning as much as I can about our systems. At this point I became the senior engineer on our team.
February 2016: ???
March 2016: Cooked up a KILLER lasagna, and went hiking in Marin. My mother and sister came down to visit at the end of the month for her spring break, and we went to KitTea, a cat cafe in SF. ALSO, I got accepted into SFSU for the fall, with graduation at the end of the year!
April 2016: Four months late, we finally had a housewarming party! I make approximately one metric crap-ton of pork ribs.
May 2016: My girlfriend finished her PhD!!!! To celebrate, we took a trip to Monterey over the weekend.
June 2016: As part of the after-celebration, we visited Universal Studios in LA for the new Hogwarts attraction, and also toured WB Studios, which ended up being really neat (walking through the Gilmore Girls’ house?!) and the highlight of the trip for me.
July 2016: At the end of June, we flew to Ireland as part of the after-after celebration. We drove around the country over the course of a week, drinking in the lush countryside and terrifyingly narrow roads. Also, Irish breakfasts are a serious reason to consider Irish citizenship.
August 2016: I handed in my resignation at Curbside; after a glorious year and a half, it was time for me to go back to school.
September 2016: ANOTHER SF big book sale! Plus, we got to go see Darren Criss and Lena Hall perform Hedwig as an early birthday present, which was, just, incredible. This was a great way to start off our year of theatre.
October 2016: We went belatedly apple-picking at an organic apple orchard. Got to see a huge old tortoise walking around, so it was 100% worth it.
November 2016: For my birthday, I was treated to Fogo de Chao, plus super-fancy high tea at a cafe. What more could anyone want? We also cooked up a huge Thanksgiving dinner, complete with whole duck and giant pumpkin pie.
December 2016: In November, after three months of unsuccessful attempts, I finally scored us some tickets to a live taping of the Big Bang Theory, so we flew down to LA this month, got to watch the actors be their amazing selves, and took a bus back to SF all in the span of a day. One really exhausting day. I visit family in MN for the holidays, this time by myself.

That, I think, brings us up to today. It’s a lot of stuff, I know, and I glossed over a whole bunch of important things, some of which I’ll write dedicated posts for. And I still haven’t completed the story of how our family got to August 2014! But I think it’s helpful to have this timeline here, just so there’s a record of the broad-strokes history.

And We're Back!

Thu, 12 Jan 2017 11:01:31 -0800

The blog has been down for a ~long~ time, as you may have noticed. Most of my personal sites have fallen into various states of disrepair, and I migrated things over to a new server, which took everything down permanently until I found the time to restore things.

But now we’re back! You may still notice some broken links or what have you around the place; let me know when you discover these and I will try to fix them as soon as I can.

Our Summer Family Trip to Europe - Prelude

Mon, 07 Jul 2014 06:24:16 -0800

Our family flew to Europe this summer for a little over two weeks of vacation time. We ended up visiting Paris and three cities in Italy, for roughly three-four days apiece. It was beautiful and eye-opening, but a lot of things ended up happening, and as a result I thought it’d be a good idea for me to wait awhile before writing the trip up. The things I want to say will make a lot more sense if I recount our trip day by day in a series of posts, and I think it’ll be lighter this way, so let’s get started.

Our flight out to Paris departed on June 12, a Thursday. Since we’d be overseas for the better part of a month, everyone was bringing a lot of stuff - four suitcases’ worth, a briefcase, my mom’s handbag, and a backpack for my sister. I’d resolved to pack light for the trip, bringing just enough clothes to last, if I double-dipped, and a book. I also planned on staying up the previous night and sleeping on the plane during the day in order to combat the inevitable jet lag - I’d done this on a couple of past trips to China successfully.

In fact, my little fight with sleepiness was just about the smallest thing anyone had done up until that point to try to make the trip a success. This was the first family vacation where the planning was explicitly divided between members of the family; my mom was responsible for Florence and Rome activities, I was responsible for Venice, and at least in theory my sister was to figure out what she wanted to do in Paris, since she’d been dying to go for ages. My mother pored over travel guides for weeks, figuring out what was a must-visit and what we’d be better off avoiding. My father booked the whole thing including transportation and hotels.

It was especially important that this vacation go well, because it was probably going to be our last one. For the past few years, my dad had had issues with his partners at his workplace. Things got to the point where one of them was actively trying to get him fired, and with physicians this is surprisingly not difficult. So at the age of 55 my dad was forced to start looking at other practices. I can’t emphasize enough how difficult and soul-wrenching this process was. He did get an offer from a practice in Oregon, and naturally accepted. But moving the family cross-country in the midst of my sister’s high school career was a tough call; my parents had already been down that road with me and they felt it didn’t go so well, so they wanted to give my sister a say in the matter. They gave her the option of splitting the family, with my dad moving and my mother staying behind. My sister, having spent fifteen years with a father who took care of her basic needs but who was completely absent from her day-to-day life and her support network, took it. I think her exact words to my father were “You can do what you want, but I’m not moving.”

Needless to say this didn’t go over so well. The discussion where this happened involved my sister breaking into tears and talking about wanting to kill herself, refusing to hug my dad, and my dad calling himself a failure of a human being.

One of the worst parts about the situation was that the discussion didn’t end there, despite the presumed finality of the decision going into the discussion. My dad continued, over the course of weeks, to pressure my sister to change her mind. I can’t really blame him; he’s looking at an incredibly shitty situation where, honestly, I’m not sure he’ll be able to survive for long without the family formally breaking up. This also heightened my dad’s sense that the work he puts into supporting the family has gone largely unappreciated, something that’d manifest in several ways over the course of the trip.

Anyways, it was super important that the trip go well. We all knew that this was going to be the last hurrah. So on Thursday, June 12, I left, sleep-deprived and running largely on coffee fumes, with my family for the airport to fly to Paris.

On the Supposed Conflict Between Individual Rights and Communities

Fri, 31 Jan 2014 21:15:43 -0800

As part of what is hopefully my last term of my undergraduate education, I’m taking a political philosophy course. As part of the general outline of the course, our professor introduced the broad overarching conflict that we’d be studying in several works, one that he phrased as Individual Rights versus Community. The idea, then, being that every individual person wants to preserve their own rights and freedom of action, but at the same time every individual person is part of a larger community that must enforce certain rules (and therefore prohibit certain things, restricting individual rights) in order to enable these individuals to live together. And in fact, this is how the issue is usually framed, as a conflict between what you’d like to be able to do and what you have to do. You see this everywhere: one of the fundamental philosophical differences between the Democratic and Republican parties of today is that the former believes that the existence of the community or society is paramount, and that society itself is responsible for most of the increases in welfare in the past century or so, while the latter emphasizes the role of individuals overcoming whatever societal hurdles are in their way to become great people. In one case, society as a whole is a benefactor of individuals; in the other, society is something to be overcome and then bettered by individuals.

For a long time now I’ve been vaguely uncomfortable about accepting this framing of the issue; I think I’m able to coherently explain why now, after some thought. I don’t actually believe that there’s a “conflict” between individual rights and the community; instead, I think what we’ve been pursuing all along is the maximization of freedoms. Let me explain with an analogy. Clearly, there are some rights which we think humans ought to have, and a bunch of these rights are pretty much unrelated to one another. Take, for instance, a person’s right to bear arms and a person’s right to trial by an impartial jury. Both of these are rights that were, at least one point, thought to be necessary for a healthy society, and they don’t have much to do with one another. Very loosely, we can say that these are “orthogonal rights” (meaning they don’t have anything in common), similar to orthogonal vectors in a vector space, and that they span a subset of some “space of rights” that we’d like to completely span (e.g. give everyone the freedom to everything).

So let’s say our goal is to have the “free-est” society possible, where everyone can do anything. How do we go about establishing this? Well, the naive solution is to simply say “everyone is free to do everything”. That should do it, right? Well, not quite! Very naturally, certain people will be more-capable than others, whether this means they’re born into wealthy families, or have more access to educational resources, or they have a genetic predisposition for physical strength etc. So if we simply said “everyone can do anything”, we’d undoubtedly end up with situations where in name someone might have the freedom from oppression by others, but they nevertheless find themselves a slave. So it turns out that the “trivial” solution (add every “rights vector” to our set) doesn’t actually achieve our intended goal (span the rights-space). In fact, I think it’s pretty clear that you cannot in any meaningful way span the rights-space; surely we cannot give people the freedom to oppress others if we are to have a free society, but prohibiting oppression in and of itself is restricting an individual’s freedoms.

What to do, then? All that’s left is to try to find a set of guaranteed rights that corresponds to a maximal spanning of the rights-space. And it is this struggle, I think, that is at the heart of what we see in American politics today. Democrats are generally in favour of gun control regulation in the pursuit of freedom from being shot; Republicans are generally against it in the pursuit of unrestricted freedom to bear arms. Democrats are generally in favour of carbon emissions regulations in the pursuit of freedom to exist several centuries in the future; Republicans are generally against it in the pursuit of *unrestricted *freedom to emit pollution in the name of economic growth. Democrats are generally in favour of regulation and tight control over the financial sector in the pursuit of freedom from corruption and abuse by the financial sector; Republicans are generally against it in the pursuit of unrestricted freedom to pursue financial innovations. In any of these cases, picking one side over the other necessarily restricts some freedoms in the name of enabling others. It’s not about “individuals” struggling to maintain their personal freedoms against society’s restrictions, and it’s not about “irresponsible” people refusing to “behave” in society.

It’s about maximizing the freedoms that everyone has.

What's Been Going On

Fri, 10 May 2013 15:00:43 -0800

So people who’ve known me for awhile know that I’ve taken a long time to finish undergrad - this spring marks my 12th quarter in my sixth year at the University of Chicago, and along the way I’ve taken two leaves of absence spanning a total of more than a year to work and figure out what’s wrong. Last year I finally admitted that there was something going on that I might not be able to handle on my own, so I started talking to a psychiatrist downtown to work through my problems. I know that there are some people who are still legitimately confused as to what’s been going on with me the past few years, so I wanted to take the opportunity, now that I think I’ve got a proper handle on what the issues are, to explain. This is going to be a long post - I’m going to be fleshing out my own thoughts as I go along, so please have patience with me.

And just a note: I want to make it clear that I’m not trying to blame anyone for anything here. Especially not my parents, who’ve sacrificed an incredible amount so that I could have all the opportunities I’ve had. Things don’t always happen the way we intend and I love the hell out of my parents for doing what they thought was right every step of the way.

To be short: on a visceral level, I don’t feel like I belong here. Well, the problem goes further than that - I think I’ve always felt distinctly out of place no matter where I’ve been, and this has weighed really heavily on me. When I was a kid growing up, I didn’t really have that much in the way of a support network at home. My parents, as first-generation Chinese immigrants, really couldn’t help me with my problems, whether they be about schoolwork or social life. My dad, as a training physician, was out pretty much all day and night, and my mother – though not for lack of trying – wasn’t able to help out with any problems I was having. I don’t think I had a substantive conversation with my parents for something like the first fifteen years of my life, after which puberty and hormones basically forced the issue.

Among the friends I made, I also felt out of place – fellow ABCs can probably relate to this. It’s kind of cliche, but at least among the Chinese people I’ve talked to there is a constant awareness that you just don’t belong in the US. My dad’s a prime example of this - even as a well-established physician, firmly in the top 1% of American families, he recognizes that if business were to go south at his hospital, as the only non-Caucasian physician in his department, he’d probably be first on the chopping block – even though his performance has been absolutely stellar. He looks different, he acts different, he speaks different, and if he’s ever disagreed with his coworkers, that just highlights how different he is. There’s a sad sort of logic to it – even though there very well may not be any conscious racism going on. So even at school, even when you’re just hanging out with your friends, you’re absolutely conscious of the fact that you’re in a very real sense on the outside looking in.

Our family history hasn’t helped this, either. We’ve moved around every four years or so for my father’s physician training, and I just had to get used to the idea that I’d only know people for a couple years before having to pack up, leave, and never look back. It slowly got to the point where I actively avoided talking to people I used to know from previous states; talking about the old days just made it all the clearer to me that I wasn’t really an important part of that world and that the place I was living now was exactly the same. If you’ve ever gone back to an old home like I have, I imagine you sort of understand what I mean - life goes on without you no matter where you were or who you are, and in a way that’s depressing to think about. So I developed this habit of just dropping everyone and everything I’d known every four years and starting from scratch. In retrospect it was really unhealthy, but it helped me cope at the time.

Fast forward to college. I matriculated wanting to pursue physics – but went through a real shock as I realised that I really didn’t know shit in many ways. I’d taken some “high level” math classes intended for graduates at the UofM in my senior year, but in reality I hadn’t learned much of substance and had spent most of it goofing off. So when my maths placement exam results came back, it shouldn’t have surprised me that I got placed into a class designed for non-maths majors. It was a huge blow to my self-confidence, though, and I immediately lobbied to be allowed to enroll in the hardest maths class I could take, Honors Calculus 16100 (IBL). I didn’t get in until after the first week of classes, but I felt pretty confident that I could catch up.

But I was wrong. Everyone in that class was clearly much smarter than me, had more background than me, was much faster than me. In retrospect this makes sense - I was doing pre-med stuff and a lot of other things, and I wasn’t able to focus on math exclusively like a lot of these other kids were – but at the time I hated myself for not being as good as my fellow students. I’d stare at assigned text for hours, struggling to parse a single sentence and feeling subhuman the entire time. It got to the point where I felt guilty just showing up to class; as if by attending I was dragging the class down and lowering the level of discourse. I fell into a downwards spiral, not doing well on exams or assignments, and this deterioration culminated in a moment that I remember pretty vividly. One of the TAs for the course asked to see me outside of class, and I went to meet him in his office in the basement of the maths building. When I sat down, he started out by mentioning that I’d fallen behind in the class, and then he straight-up asked me “Why are you in this class?”

If I wasn’t feeling bad beforehand, at this point I sure as hell was. It really shook me; all these feelings of failing not only the class but myself and my parents, of feeling like I didn’t belong where I was despite fighting to get there, of feeling like I couldn’t ask anybody for help and that I was a failure for not fixing things myself (I’ve had it drilled into me from childhood that self-reliance is a gold standard), coalesced and I pretty much broke down right there. It wasn’t pretty and I think the TA wasn’t expecting this. The rest of the conversation is largely a blur.

After that, I had serious problems attending classes. I mean, I’d had problems beforehand that had built up over time, but after that point things never got any better. I’d go to classes for the first week, internally fighting this feeling of not belonging and trying my hardest not to screw up, feeling like I was lying to myself and other people by pretending I was good enough to be there, and inexorably something tiny would eventually go wrong and I’d withdraw into a shell, not going to classes, not talking to anybody, not sleeping or eating for days. And then because I hadn’t eaten or slept for days, I’d eventually crack and eat and sleep for an entire day or two. There were days where I wouldn’t leave my bed for anything, or days where I’d starve myself. I think I was punishing myself for screwing up. I know it doesn’t make sense – by punishing myself I was screwing up even more – but at the time I really didn’t know what else to do, I felt so awful.

Around this time I also developed a major complex about burdening other people; I’d go way out of my way (and I still do, to some extent) to avoid making people feel obligated to do anything for me. I think my reasoning was that I was just such a shitty person that it’d be ridiculous of me to expect people to do things for me. It got in the way of *everything - I’d agonize and procrastinate for hours over having to send an email to a professor asking to get into a class, or actively avoid study groups with fellow students because of course I wouldn’t have anything to contribute since everyone I knew was so much smarter than me.*

It was really terrible. And I was stuck in that awful place for years. I couldn’t talk to anybody about what was going on because I was so ashamed. I couldn’t get help because I didn’t want to admit that there was something going on with me that I needed someone else’s help with. And through it all there was this feeling of being totally alone, that if I told anyone what was going on I’d be screwing up in the worst possible way since it’d be burdening them with my insignificant problems, problems I really even shouldn’t be having since I’ve been given so much by my parents already!

I mean, seriously. What did I really have to complain about? My family’s really well-off and all things considered I’ve avoided pretty much every childhood trauma aside from parents having marital problems – but marital problems are pretty common nowadays anyways. I should be sailing through life with all the opportunities I’ve been given. All the more reason to feel shitty and guilty about even having problems in the first place.

Eventually things came to a head, my parents found out, and I went home for awhile to sort things out and try to piece things back together. This actually happened twice. The first time around, my dad insisted that I just needed to find some work and spend some time away from classes - which, after a couple agonizing months of job searching while my dad guilt-tripped me for not having found a job yet, sort of worked for awhile, but when I went back to school I felt just as shitty as I had before, and the same thing happened. The second time I managed to convince my parents that I needed to talk to someone, so I met with a psychiatrist at the UofM who diagnosed me as having had pretty severe depression.

So I moved back to Chicago to work a research job for awhile and got referred to a psychiatrist downtown, and I’ve been talking with her for the past year now and taking some medication. Things haven’t always been good, but they’ve definitely gotten better; I was able to hold a job pretty consistently for a year, and last quarter I was able to finish two classes. Another thing that really helped was running; I started running outdoors over the summer, and I can’t begin to describe how much better I feel when I’ve run a couple miles.

So that’s where things are. I’m not fantastic yet. But things are getting there. I’m not really sure how I’m going to tackle this ’not-belonging’ thing, but at least I know what it is and I’ve started to improve on the other things. Thanks for reading this far; if you’re reading this, you’re probably a friend who’s had their patience tried by my shit in the past few years and I haven’t, up until now, known how to explain it to you. I really appreciate it.

Forecasting Using Seasonal Adjustment Factors

Sun, 14 Apr 2013 14:57:29 -0800

I just happened upon Analytics Made Skeezy, a blog written by John Foreman, Chief Scientist at MailChimp. He posts every so often detailing various techniques used in analytics, ranging from revenue management using price elasticity estimation to clustering and community detection. But there’s a twist - all of the posts are written from the perspective of a (presumably fictional) college student who finds himself working business analytics for an international druglord! It’s very much a Breaking Bad-esque take on what is already becoming a very sexy field. Go take a look, it’s well worth spending an afternoon on if you’re at all interested in the topic.

Anyways, I was reading through the posts and I found myself reading the entry on forecasting demand for drugs based on prior monthly data. Our hapless college student goes through the motions, explaining how to use what are called seasonal adjustment factors to nudge an otherwise-linear regression upwards and downwards to reflect seasonal fluctuation in demand. What really struck me was that this is almost exactly the method I independently worked out a year or two ago on End of the Internet, forecasting user activity numbers from historical data. I started doing this analysis after some major changes were made to the structure of the site and incentives for users to participate, to see how they’d affected the community as a whole.

What I did then was a little more elementary with regards to smoothing, and I used a couple of different variables to generate my projection, but the basic method remains the same - for each year, you figure out how much each month in that year deviated from the average value of your metric for that year. Then you either:

average all of these monthly deviations for Jan, Feb, Mar, etc. individually to obtain your seasonal adjustment factors, or
perform some sort of regression on each month’s yearly deviations if you believe that the factors are changing over time.

From there, you can use a simple recursive multiplicative model using the prior month’s data and your seasonal adjustment factors to obtain any future month’s predicted value.

Back when I was doing this for End of the Internet, it was mostly a passing curiosity - I tracked numbers for a few months using the python daemon I’d already written and wrote a post each month with the new numbers and revisions to the model. I went back today and compared the performance of my (more elementary) ETIStats model to the exponential smoothing model described by John for the year of 2011:

As you can see, exponential smoothing outperforms my original model; the RMSE is 67022.55, as opposed to ETIStats’s RMSE of 91581.66. Plus, the variation in monthly error seems to be a lot lower for exponential smoothing; the variance in the absolute percent error is almost half that of my original model. Still, the general trends seem to remain largely intact, and given the noisy data I had at the time for the other variables I was tracking, I’m pretty pleased with my original results!

Spring Break 2013 in Chile

Sat, 13 Apr 2013 18:58:52 -0800

Well this is a few weeks late! My father and I went to Chile for my spring break for about a week - between March 22nd and March 29th - to go hiking in Torres del Paine National Park. We both had a lot of fun - normally when our whole family goes, there’s a lot of planning and extensive packing ahead of time, but since it was just the two of us, it was a lot more ad-hoc. Throughout the trip we took a ton of photos and I kept some notes, so here’s a bit of a writeup of our vacation!

Day 1 (This post)
Day 2
Day 3-4
Day 5
Day 6-7

Day 1 - Santiago

I headed up to O’Hare the afternoon of the 22nd, straight after my last final for the winter (whoo). Genius that I am, I’d pulled an all-nighter the night before studying for it. My dad was flying in from MSP, so we met up there and headed to our connection through ATL, which went off without a hitch. I knew that ATL was the busiest airport in the world, but it hadn’t really sunk in; I was pretty surprised at how much nicer it was than O’Hare - and I’m pretty fond of O’Hare! Anyways, we arrived at ATL in the evening, grabbed dinner really quick, and got on our flight to Santiago, the capital of Chile. Ten (largely sleepless) hours later, we were in Chile!

When we got in, I immediately knew we were in for a bit of a rough trip - I “studied” Spanish for almost five years in middle and high school, but never really excelled at it, and I hadn’t used it since. So my speaking skills are basically zero, and my reading and writing skills are– well, I could probably pass for an elementary-school kid if you didn’t grill me too hard. And my dad doesn’t know a lick of Spanish! So we (I) could’ve prepared better there.

Upon arrival we were required to pay what the Chilean government calls a “reciprocity fee” of $160 per person, levied on US tourists. This was really weird to me at the time, but apparently it’s a fee that was implemented in retaliation for the US government charging Chilean nationals to process their passports upon arrival in the US. But the money doesn’t go to reimburse said nationals - it goes straight into the government’s coffers, so it’s really a bizarre thing.

Anyways, we passed through customs without incident. My dad tried to purchase a SIM card so we could make and recieve calls if need be, but when we tested the card out at the vendor, the phone would just shut off – apparently during the trip it’d gotten turned on somehow, and the battery was completely drained. Rather than run the risk of buying a card that didn’t work, we decided to forego a phone altogether, so this is another area where we could easily have prepared better.

After that, we took a taxi downtown to our hotel, which ran us about 15k Chilean pesos – for reference, the exchange rate at the time was about 470 pesos to the dollar, so roughly speaking, 10k Chilean pesos were equivalent to $20. On the way there, I was struck by how similar everything felt to cities I’d been to in China – the outlying areas around the city were very undeveloped, with the occasional shack or dirty apartment building here and there, but once you got into the city it was high rises and skyscrapers everywhere. Santiago very much felt like a city that had gone through rapid modernisation recently. Pollution was pretty bad, though I didn’t really notice it much, and there was graffiti everywhere; almost literally every building’s ground floor walls were covered in spray paint.

The first night, we stayed at Chile Apart - the room was really nice, though there was no central cooling or heat. We had a fully-equipped kitchen, living room, and bedroom for an extraordinarily reasonable rate by US standards.

We dropped by the neighboring supermarket – which turned out to be owned by WalMart!! – to pick up some supplies for the trip ahead, and went to the Plaza de Armas at the center of downtown:

There we visited the Cathedral of Santo Domingo, which was just absolutely gorgeous. I’m not really much of a religious person – and the Catholic church has played a key role in how conservative Chile is today – but I can definitely appreciate works of art like this:

After that, we headed for Santa Lucia Hill, a park and viewpoint in the middle of the city. There was a castle of sorts at the top, and we got to the top just as the park was closing, so we didn’t get that many photos, unfortunately. It also turns out that the hill is a favourite destination of couples, so we had a couple of awkward moments where we had to walk around people making out in the park. Still, the sunset was gorgeous and it was a nice way to start off the trip.

Then we headed out to grab dinner. We wandered a little around the city, being pretty indecisive at first, but eventually we decided to stop at an upscale Italian restaurant called Nolita. Since it was the first day, we splurged and ordered an appetizer of a dozen raw oysters, with our entree being this massive slate of grilled seafood:

Needless to say, we had a really good first night in Chile, despite the fact that I had been, at this point, awake for about 48 hours. Oh, and on the way back, we saw this:

The next post will continue with day two, where we flew to Puntas Arenas and spent four hours getting lost driving despite there being only one major highway in the region. Stay tuned!

Bit of an update on Animurecs

Sat, 13 Apr 2013 00:27:56 -0800

It’s been a long time since I talked about this, but I’ve been re-writing Animurecs in PHP, having gotten fed up with feeling like I was working around Rails to get things done. Wasn’t an easy decision - there was a lot of work already done, and I hadn’t written an MVC framework before. I’ve had a lot of fun, though, and I feel like my programming skills have benefited from it! The project has evolved from originally a Danbooru-esque anime database to a social network that uses machine learning (largely based off of Simon Funk’s explanations of SVD) to figure out what each user’s tastes are and recommend anime they’d like based on that. There’s also tons of unexplored ways to encourage people to watch anime together and discuss it in some depth; I’m hoping to encourage that with this project!

On that note, it’s reached a point where I’m pretty comfortable putting it into beta status – you can find it here. After registering and confirming your email, you can add a few entries to your anime list – the SVD is recalculated every hour on the hour, so you should see recommendations popping up for you then. You can also add friends on the site, if they’re registered, and the site will figure out potential groupwatches you can join. Check it out!

Finally, the site is under pretty heavy development, so you should expect to encounter a few rough edges here and there. If and when you run into bugs, you should feel free to contact me via Twitter or just post on my Animurecs profile!

ECLS-K: A General Overview

Mon, 14 Jan 2013 00:09:42 -0800

So I’m taking classes this winter and spring; finally on track to finish my BA in math. One of the classes I’m taking is an intro sociology class - one of the overarching goals of the course is to understand the interplay between parenting and education, and how these two factors help children become successful later in life. As part of this investigation, we’re using data gathered by the Early Childhood Longitudinal Study, Kindergarten cohort (or ECLS-K). This was a nationwide longitudinal study following a nationally representative sample of 21,000 kindergartners entering in 1998-1999. Unfortunately, I don’t have access to every feature in the dataset - I get the feeling that it’d be difficult to know where to start if I did - so we’ll be looking at a slice in our class.

Since we’ll be working closely with this particular dataset, I figured it’d be interesting and useful if I did a series of posts looking at features and talking about what we can infer from the data. I don’t have a whole lot of experience with education-related research; I followed the LA Times’s value-added analysis of teachers in the LA school district with some interest, but nothing really beyond that. It’s pretty incredible that a study of this magnitude was able to be done at all, and it’s kind of a shame that it’s not done more frequently, given how (maybe) important K-12 education is to kids’ futures. In addition, this will give me an excuse to brush up on my R/ggplot2 skills and (maybe) post some of my scripts - hopefully they’ll be of use to people looking to learn.

In any case, let’s get to it! Let’s start off by getting a general idea of what the sample of kids we’ll be looking at is like. Just a couple of bar charts for now:

Nothing terribly unexpected here; the US national male/female ratio is slightly under 1, so as far as a nationally-representative sample we’ve got some over-representation of males here. Looking at race, white non-hispanics make up approximately 56% of the sample and comprised 69% of the population in 2000. Black non-hispanics make up ~15% of our sample and ~12% of the 2000 population, while all hispanics make up ~18% of our sample and ~13% of the 2000 population, so we’re over-sampling blacks and hispanics relative to white non-hispanics. Asians make up 6% of our sample, while they comprised 4% of the population, so in general we’re over-sampling minorities.

Looking at the age each child entered kindergarten, we see pretty normal results - a bell-shaped distribution centered around ~65.5 months, or ~5.5 years old. Makes sense, since kids enter kindergarten around age 5-6! The standard deviation of this sample is ~4.30, and you can tell just from looking that the vast majority of the responses fall between 60 and 72 months old. It’s pretty interesting just how tight this distribution is - everyone falls between 4.5 and 6.7 years old, which I wouldn’t have expected, since I remember there being some much older (and really young) kids in my classes later on. It’s pretty cool to see parents valuing education so highly!

Finally, a look at parental expectations for their kids. It’s clear that college is the absolute gold standard in the US in terms of education; failing to complete a four-year degree is pretty clearly failing the expectations of your average parent (with ~41% of the responses). I honestly expected high school graduation to have a stronger showing here; I was under the impression that there was a much larger population that didn’t particularly value undergraduate education, but it’s clear that I was wrong. I feel like the proportion of parents expecting their kids to attain a post-graduate degree (~24%) is way out of proportion to what actually happens.

It feels weird having my preconceptions of our country’s population corrected so significantly. Turns out I had an overly-pessimistic view of education in a couple of areas, so it’ll be interesting to see if this continues to be true. Next time we’ll look a little closer at these kids’ environments to get a clearer picture of how parenting in the US is “normally” done!

Detecting Subcommunities Within Social Networks

Sun, 26 Aug 2012 02:03:46 -0800

I’ve been an active member of an online social forum called End of the Internet for a few years now. The site is particularly interesting because it’s both closed off to the public and has been relatively long-lived despite the lack of new users; the active posters on the site have been around since the site’s founding in early 2004, and watching the topics of discussion shift as the userbase grows and ages has been really interesting.

I’ve always thought that it would be really fun to generate a map of the social networks on the site, and in recent years the rise of niche-interest megatopics has been of particular concern to the site staff. For a long time, I’ve been trying to find a metric that accurately describes the distance between users on this forum, and I think I’ve finally hit upon something that gives sensible results.

First, I take the postcounts for a pair of users in all topics in which at least one of them has posted.
If the number of topics in which both of them have posted is less than some threshold (20), I throw this pair of users out and move on to the next pair.
Then I calculate Pearson’s r (correlation coefficient) for this two-dimensional array of post counts. If it’s above a certain level (0.2), then I place an edge connecting these two users, weighted by the value of Pearson’s r.

Afterwards, I’ve got a weighted map that I toss into Gephi and run Forceatlas2 on for awhile to let the map come to equilibrium. Then I run the default clustering algorithm in Gephi to assign each node to a community, and do a few small coloring/labeling touchups to make the graph look nice. I throw the resultant gexf file into SigmaJS and get something like this:

Pretty cool, huh? Each of these clusters corresponds to an actual niche on the site, which was really cool to see just fall out of the data. You can see an interactive version of the map here. It’s really interesting to see the degree to which the site has split into niche topics, and in particular it’s kind of scary to see just how little each of the largest clusters links with the rest of the map.

There are some issues with the metric I’m using; for one, it doesn’t work particularly well for users that don’t post in any particular niche megatopics, but nevertheless do have some friendships with users on the site. I’ll see if I can’t figure out a way to incorporate them better into the network. In the meanwhile, I’m pretty satisfied with how things turned out, and I’m in the process of generating similar graphs for previous years.

Anything Can Be Made to Sound Silly

Sun, 13 May 2012 00:41:24 -0800

Representative Jeff Flake (R-Ariz) led a House vote on Thursday to pass an amendment prohibiting the NSF from funding political science research. His primary reason for prohibiting this funding is largely out of concern that the current budgetary climate makes it irresponsible to spend this money on something that has, in his eyes, little chance of producing important results:

So what kind of research is NSF charging to our credit card? $700,000 to develop a new model for international climate change analysis; $600,000 to try to figure out if policymakers actually do what citizens want them to do.Let me say that again: $600,000 here spent trying to figure out if policymakers actually do what citizens want them to do. I think we can answer that question in about 5 minutes when we vote on this amendment because I can tell you, people out there want us to quit funding projects like this.

This amendment seems to me to be ill-motivated, for two reasons. First, if there is a problem in the field of political science with merit-less studies being funded beyond reasonable levels, the solution to this problem is clearly to improve the process by which grants are awarded to researchers! Banning the NSF from funding any political science in order to keep silly studies from being funded is akin to banning all alcohol production in order to prevent minors from obtaining it; it’s clearly an overreaction and may very well cause more grief than the proposed “wasteful” studies would have caused. What’s more, the actual amount of funding provided to political science by the NSF is a tiny drop in the bucket in the scope of our national budget! Eliminating this does essentially nothing to moving us closer to an annual budget surplus, and there are far larger and more-bloated departments (Defense, anyone?) that could easily absorb massive cuts.

Second, one of the most important hallmarks of an intelligent, mature human being is the recognition that things that aren’t interesting to you may still have significant merit and importance! Most people are bored to tears by physics and biology, but these fields are incredibly productive and the fruits of these fields are in many ways the foundations of our modern society. It’s incredibly easy to find studies that at first glance sound silly in any field, too. Tom Coburn previously railed against the NSF for funding shrimp treadmills as a total waste of money, but it turns out that the treadmills were a tiny (actually maybe free) part of a much larger study on the ways in which shrimp react to changes in their habitats. When this distortion was discovered by NPR, a spokesperson for Coburn replied:

Our report never claimed all the money was spent on shrimp on a treadmill. The scientists doth protest too much. Receiving federal funds is a privilege, not a right. If they don’t want their funding scrutinized, don’t ask.

Do I really need to say more? It’s very easy to make something sound silly, especially in fields that aren’t well-understood by the public. In a way, this is a big reason why it’s important that people like Brian Greene and Michio Kaku do the work they do - without people like them to popularize otherwise hard-to-understand fields, they would certainly be targeted by attacks like this. It’s unfortunate that political science has been targeted in this way, and it’s always disappointing to see a Congressperson fly the banner of anti-intellectualism in such a blatant fashion.

Budgets and Rational Allocation

Sat, 12 May 2012 00:22:32 -0800

Suzy Khimm ran an article today about an interesting study done by the Center for Public Integrity showing that, contrary to commonly-held beliefs about party values and lines, Democrat and Republican citizens alike would choose to cut huge swaths of the defense budget:

The author of the study attributes this to the increased information and flexibility afforded to study participants; when people are made aware of the diversity of spending within the military and the levels of spending within the military, they are much more receptive to the idea of budget cuts than if they mentally conceive of the spending as one monolithic category. While I’d love to see an actual comparison between “less information” and “more information” to justify this claim, this makes a hell of a lot of sense - it’s the reason that insurance companies were so resistant to the idea of “nutrition labels” for health insurance plans, which would outline plainly what each insurance plan would cover and to what degree. It’s also, I think, part of the reason why Obama made a push to provide taxpayers with “tax receipts”, which list by program where your individual tax dollars go. It also has the added benefit of making you feel more personally-responsible for the good (and bad) things that your government does! And I know that we haven’t seen the last of the idea at work here - there’s huge incentives for providing the average citizen with more information breaking down how their money or energy is spent, and fundamentally-speaking I feel that we’ll need to better-educate the general public on these matters if we are to expect them to more effectively hold our government and politicians responsible for their actions.

(On a totally unrelated note, I’ve recently rekindled my love for Mint.com. Their web interface is pretty fantastic!)

Links

The trouble with rules - I was once a moderator on a website that faced this exact same issue; under some pressure from one of our staff members, we revised the community guidelines to replace general principles with more-specific rules. What we found was that this just incentivized people to find edge cases and loopholes, and then cry foul when we punished them anyways. In the end, the move lost us a lot of respect from the community and we moved back to general principles.

West Virginia paid $22k for each router it installed in rural areas - It’s tempting to call this a cut-and-dry case of a government official cutting a major corporation a sweet deal, but the truth of the matter is probably much more depressing (the official making the call probably just doesn’t understand technology at all).

Transfiguration MN has a new domain name, thank god - How did this even happen?!

A Brief Status Update on Where I'm At

Fri, 11 May 2012 00:00:23 -0800

I’m currently with my parents for the weekend, and things have been really busy. As a result I haven’t had the time to write this post I’ve been turning over in my head; trust me, though, if we’re at all acquainted it’s worth the wait. I’ll be here until Sunday, after which I’ll be headed back to Chicago and should be able to give this post the thought it deserves. Tomorrow and Sunday, however, I’ll probably run shorter policy-related posts as I feel like I’ve focused a bit too much on communication recently.

After that, I start work! I’m really excited to use web2py on a formal basis and it’ll be my second MVC framework after Rails, so hopefully the relationship will be a beneficial one. I’ve also been looking forward to just working in general as I’ve been out of the job market for almost a year now. Starting to get the itch to do something actually productive, y’know?

In any case, here’s the links for today! Please look forward to the posts in the coming days.

Links

A status update for sk-learn - This is, in my opinion, one of the best machine learning libraries and toolkits out there. It’s true that R is incredibly flexible, but the ease with which you can begin using sk-learn and iterating rapidly just blows any alternatives out of the water for me. To hear that major additions are in the pipeline is great.

How to serve 150 free meals for less than 20 cents each - Bravo to Food not Bombs for doing this; this is exactly the sort of innovation and practice that we need to see happen more frequently if we’re to begin to reduce poverty.

The emergency room is not health insurance - Sarah Kliff does some excellent reporting on the limits of so-called “free healthcare”. The idea that the plight of the uninsured is mitigated significantly because of the federally-mandated obligations that hospitals have to provide a bare minimum of care always rung hollow to me, especially now that we’re hearing more frequently about “collection enforcement” picking up in hospitals.

Treating Your Opponents Like Human Beings

Thu, 10 May 2012 00:37:56 -0800

I recently came across Science-Based Medicine, which is an absolute treasure trove of articles about a plethora of topics in medicine written by an all-star lineup of physicians in a variety of fields. I have immense respect for physicians who take time out of their very busy days to try to educate the general public and contribute their expertise and knowledge to the public discourse; having lived with a physician for essentially my entire life, I understand just how difficult it is for someone who comes home late at night, mentally and physically drained, to muster the willpower to spend even more time delivering what amounts to a free consultation worth thousands of dollars to the internet. To do this on a regular basis really is something that ought to be commended more often, so it’s fantastic that SBM has managed to pull together such a diverse group of people and keep them writing on such a consistent basis.

That being said, one particularly recent post on SBM caught my eye: Dr. David Gorski recently authored a defense of plausibility bias in medicine in an article entitled “Plausibility bias? You say that as though that were a bad thing!”. In this defense, Dr. Gorski responds to an article published in Medicine, Health Care, and Philosophy that attacked the medical profession as being overly closed-minded with respect to “alternative medicine” such as homeopathy. I don’t disagree with Dr. Gorski’s rationalization of plausibility bias in medicine and science in general; the very goal of the scientific process is to arrive at the mathematical model of the universe that has the highest-possible statistical confidence level of perfectly representing reality. It then makes perfect sense to impose increasingly-stringent requirements on new models as our incumbent models accurately represent larger swaths of the physical universe. “Plausibility bias” is a negatively-framed way of saying, as Dr. Gorski puts it, “extraordinary claims require extraordinary evidence.”

It does, however, seem to me that Dr. Gorski has lost sight of what his goal as a physician speaking on medicine in a public forum should be; his opening bullfighting metaphor seems to me to be especially ironic here. As a physician, his goal ought to be to improve public health. As a scientist, his goal ought to be to improve the public understanding of science. As someone speaking in public who sincerely believes in the correctness of his cause, his goal ought to be promotion of an open, honest environment in which the truth of the matter discussed is laid bare for all to see. Instead, we see Dr. Gorski embark upon a smear campaign, attacking Rutten personally as “so ‘open-minded’ that your brains threaten to fall out” and “nutty”. His tone throughout the piece comes off as condescending and vitrolic, which serves his (otherwise-accurate) assessment of the flaws of homeopathy very poorly. I mean, seriously, look at this:

That’s my plausibility bias. I’m biased in favor of science and reason and against magical thinking like homeopathy and reiki. You should be biased too.After I had stopped laughing in response to seeing homeopaths lecture scientists on what is and is not scientific, I delved into the paper. Rutten et al try (and fail—after all they are homeopaths) to establish their scientific bona fides righ in the second paragraph: [Emphasis added]

Why is this necessary? Is the article really any better off for it? Plus, who exactly is Dr. Gorski writing for here? Clearly practicing homeopaths would be strongly offended by the insults in this article, and somehow I doubt that he’s doing this to provide entertainment for those of us who already don’t believe in homeopathy. Drowning what could be a productive, enlightening conversation in personal attacks and condescension does his entire profession and field of study a disservice. The very next sentence Dr. Gorski writes gets to the heart of the matter:

My first temptation was to point out that the very fact that they are homeopaths means that they are either deluding themselves or lying when they claim that they do not reject any part of the naturalistic outlook. [Emphasis added]

Obviously the vast majority of homeopaths don’t actively live massive lies that they’re consciously aware of! What’s going on here is probably a much more gray case of motivated reasoning - the same motivated reasoning that scientists also have to be paranoid about all the time! So, if your goal is to encourage people to pursue the truth, it doesn’t make sense to actively provoke homeopaths in the way that Dr. Gorski does here - since it’s clear that homeopaths are for the large part not pathological liars, it makes no sense to gleefuly ridicule them for advancing their sincere beliefs. All that will accomplish is to force homeopaths to cling even more tightly to their beliefs and become even more vocal. What does make sense is to point out the flaws with their beliefs in a way that will encourage them to re-examine their beliefs and discourage them from spreading their misinformation any further. As Ezra Klein puts it so well,

In general, I think political debates are both much more sincere and much less rational than most folks believe.

Treat human beings like human beings. Otherwise all you’re doing is making someone else’s job much harder down the line.

Links! Links for Everyone!

Mon, 07 May 2012 23:25:28 -0800

Holy god I’ve been ridiculously busy today. Believe me, there’s good reason for why there’s no blogpost today. I’ll have plenty of time either tomorrow or the day after to type up a couple to make up for it. And there’s been a lot of good stuff to write about, so please stay tuned!

The top 10 papers in Computer Science by Mendeley readership - Some of this is really surprising (LDA topping the list, for one thing). The sample sizes aren’t terribly large here, so I’d take the results (and following analysis) with a grain of salt, but undoubtedly some of these papers deserve to be on the list! I also really like the readership-field breakdown, though the charts for each paper could stand to use some work.

American Airlines’ unlimited first-class pass debacle - It’s pretty cool to hear about this sort of thing cutting the other way for once, and it’s really interesting to hear the steps that companies take to crack down on what they see as a loss-leader.

A statistician justifies his purchase of lottery tickets - I must admit, I hadn’t really thought very much about the “hours of enjoyment” angle. Though my immediate reaction to this argument is that the accumulated loss over all tickets outweigh the one or two hours of fantasizing you get in return!

Startups and the Homeless

Sun, 06 May 2012 22:00:01 -0800

So today an interesting headline passed across my feed - yesterday the Washington Post [ran an article](http://www.washingtonpost.com/business/technology/hacking-homelessness-san-franciscans-hope-startup-approach-can-ease-neighborhoods-problems/2012/05/05/gIQAS8p43T_story.html) detailing the efforts made by startups as part of a hackaton to ease the suffering of San Francisco’s homeless.

Barry Roeder, a San Francisco management consultant, wants to eliminate the lines by creating a neighborhood-wide network of touch-screen kiosks where people could make and check reservations themselves. The system could also notify people by text message if they received a bed — the Creative Currency survey found that while few residents have smartphones, about 60 percent have access to some kind of cell phone.

As well-intentioned as these efforts are, the thing that struck me while reading the article was how silly the proposals sound - while it’s amazing that these people are trying to make the plight of the homeless less-painful, there is something very misguided and wasteful about proposing that a company or city government pay for touch-screen kiosks to make the process of reserving spots in the homeless shelter less painful. The real issue at hand is that San Francisco’s homeless shelters simply don’t have enough space for the city’s homeless! It seems pretty clear to me that the solution to this problem is not to spend millions on making reservations at homeless shelters easier to make, but to expand the homeless shelters themselves or address the fundamental problems leading people into poverty in the first place.

The high-tech idea proposed in the article makes crystal clear one of the biggest reasons why you don’t see tech startups in Silicon Valley effectively addressing and attacking poverty - not only is there very little money to be made in helping poor people, the people who run and work for tech startups for the large part have no idea what the actual problems causing homelessness and poverty are, and don’t understand what the biggest issues facing a homeless person are. All this ties back into the plasticity of social norms - if you’re not constantly immersed in poverty and surrounded by it, you can’t be expected to know much about how to fix it. Not that this is an unfixable situation - but when there’s just so much money in tech right now, I can’t imagine many startups directing their efforts to this problem. It’s a damned shame, too; lack of money (and social stigma) is the problem here, after all.

Links

Three researchers from the University of Berlin quantify the relationship between unemployment and happiness - This is really interesting work, though the direction of causation still needs to be determined. I’d also be really interested in a study that looks at if there’s a relationship between the number of unemployed friends a person has and the degree to which their unemployment affects their happiness. It’s too bad that the BLS longitudinal datasets don’t track peer data. If anyone knows of a public dataset that does, I’d be happy to do a study!

There’s a weekly casual Starcraft meetup in northside Chicago - If this wasn’t an hour and a half away I’d go every week! I’m really surprised that there aren’t more major SC2 tournaments or events in Chicago given its central location.

Better Links Than Never

Sat, 05 May 2012 20:00:42 -0800

Hey everyone! Sorry about the placeholder entry earlier today; I got caught up in hanging out with my sister (and making some delicious pork ribs) and totally forgot about the blog entirely. Won’t happen ever again. I think I went a little too far yesterday with regards to length, so I’ll be doing some shorter entries to prevent myself from getting too long-winded. Since it’s so late today, I’ll just post some links and leave the meaty material for tomorrow. Apologies!

Sean Carroll explains in easy-to-understand terms the physics on why there is something rather than nothing - Every time I hear about modern physics I always come away with a deep appreciation for the work that these men and women do. In many ways I regret not studying physics in more depth at college. (h/t to Phil Plait over at Bad Astronomy for this one)

Peter Theil’s CS183: Startup class in essay form - I’ll be following this really closely. Startup life is something I’ve been pretty interested in, and it’s really rare to get a free glimpse into a veteran’s wisdom like this.

TU Munich’s Mathematics department cancels its Elsevier subscriptions - This needs to happen more often. I’m glad that Harvard put out a strongly-worded statement against the worst practices of academic journals and publishers, but in the end what’s needed to achieve real change is action to impact their bottom line.

Health Affairs outlines why drug shortages happen - The implications of how sensitive our stockpiles are to the shutdowns of individual factories are really scary.

Overconfidence and Communication

Fri, 04 May 2012 22:00:33 -0800

Today I turned in the paperwork for my job! Hopefully I’ll start soon.

Something that’s been bugging me for awhile about the communities around the hard sciences and tech fields is that very frequently, you’ll encounter people who are dead certain that their opinions are absolutely correct and that anybody who disagrees with them is not only incorrect, but also flawed as a human being for disagreeing with them. You see this all the time in software development - I’m sure you’ve met someone who would snicker at anyone who was running a non-Linux OS, or using PHP, or not camel-casing their variable names. Not that this is necessarily limited to software development: I’ve worked with people who were very vocal about the worthlessness of the social sciences relative to “harder” sciences, or the existence of a deity. It can be painful to try to [talk reasonably](http://www.codinghorror.com/blog/2008/05/php-sucks-but-it-doesnt-matter.html) with these people to get things done, since for many of these people there’s absolutely no room for compromise - their way is the One True Path and anything else is absolutely inferior and incorrect.

“Half of what we’re going to teach you is wrong - the problem is, we don’t know which half.”
Popular medical school teaching

Fundamentally, I think the issue at hand here is that people feel the need to be confident about their choices in life. We’re all taught from birth that confidence is something to be pursued and desired, and while growing up our role models constantly project an image of self-confidence and certainty. And this makes sense - as a parent, you instinctively don’t want your kids to grow up doubting and questioning themselves at every turn. That doesn’t sound like a great way to live, and if they’re ever going to be anyone of importance, it makes sense that they need to be able to convince people to follow and trust in them.

To admit even the possibility of different-but-equally-reasonable viewpoints on a topic nowadays is to fatally handicap your arguments; after all, if everyone was constantly consciously-aware of the fact that we really know very little about the things that really matter (e.g. whether or not God exists, what “best practices” really are in software development), everyone would start to more critically-assess arguments made by previously-considered authorities! It would be much more difficult for people to attract a following, where previously mere force of rhetoric was considered sufficient to attract an audience. And the truth that we really don’t know about the most important things is scary! Nobody likes dealing with uncertainty, but simply masking the truth is doing everyone a disservice, yourself included.

As a result, we’re becoming more and more “confident” - even when people disagree over something, each side is insisting all the more strongly that they are absolutely correct. But this comes off as the wrong approach to me, especially today where the problems we’re pursuing are much more complex in nature. In a society where the best representation of physical reality we’ve got is quantum physics (which continues to humble me in its complexity), I don’t think we can really afford to let the state of our public discourse degenerate so far. To do so is to simultaneously shut the door on further improvements in human communication, a field that is becoming all the more important in our more highly-interconnected society, and also resign ourselves to an increasing disconnect between the universe and our conceptions of it.

This is not to say that all confidence is unwarranted - I’d say that we understand the physical universe much better than we did a millennium ago, so to say that we understand quite a bit about how the universe works is perfectly realistic. To take Richard Dawkins’s stance on theology and actively disbelieve in a divine creator based on irrelevant scientific knowledge, however, is overconfident and arrogant, and encouraging people to behave in this way is incredibly destructive to the actual pursuit of knowledge, theological and scientific. Similarly, to shut down a startup proposal because “no one would be using it in 10 years when everyone has a free Android smartphone” is deeply misguided on many levels, not the least of which is assuming anything about the state of technology 10 years from now. If you want to have a productive conversation with someone and come away with more than just a superficial feeling of dominance, you’ve got to actively be aware of the limits of your knowledge and expertise and communicate that to your partner.

So to this end, I want to make it clear that anytime you see me being arrogant or overconfident, please call me out on it! I’m flawed in this regard, just like everyone else, but I’ve been trying to change myself for the better and I hope you’ll help me along that path.

Links

Harvard is hosting a workshop on computational social science - If I were in the neighborhood, I’d totally sign up for this. I’ve toyed around with Gephi and made some decent-looking network visualizations before, but I’ve been dying to learn from professionals.

Opinions Netflix has formed about me during my 3.2 years as a customer - Some of this is pretty misguided, but it’s a good reminder that small datasets have very limited predictive power.

Ezra Klein interviews Paul Krugman - The most interesting part of this to me is Krugman’s response to how the Greek debt crisis would look in a country that has full control over its currency. It really underscores how much of this problem is really a matter of incredibly-hazy human perceptions of how likely it is that a government will remain committed to currency stability. I can see why Bernanke is so very committed to the inflation half of the Fed’s dual mandate.

Kicking the Daily Blogging Off!

Thu, 03 May 2012 20:00:58 -0800

Today’s the first day of this daily blogging thing that I’m going to do for the conceivable future! I figure a good way to start off is to talk briefly about a topic that I’ve been thinking about each day, and then at the end I’ll post some things to interesting articles or blogposts and a brief summary of things I’ve accomplished that day. So let’s get down to business, eh?

There’s been a bit of a buzz in the last few years about the economic impact of inequality - there was a recent New York Times article detailing one man’s stance on why high levels of inequality might represent a healthy economy. Fundamentally, his argument rests on the idea that the absolute level of savings at the top 0.1% is actually pretty low, and that higher earners do a significant amount of reinvestment of their earnings back into the economy, which has significant benefits for the general public:

Conard understands that many believe that the U.S. economy currently serves the rich at the expense of everyone else. He contends that this is largely because most Americans don’t know how the economy really works — that the superrich spend only a small portion of their wealth on personal comforts; most of their money is invested in productive businesses that make life better for everyone. “Most citizens are consumers, not investors,” he told me during one of our long, occasionally contentious conversations. “They don’t recognize the benefits to consumers that come from investment.”

While it is totally true that there are incredibly wealthy people who do a lot of investment into the economy and thereby avoid a lot of the lost potential economic growth that would be associated with high levels of saving at this income bracket, I feel like this argument misses the point - I can’t see why this system is any more efficient or desirable than a society in which wealth is more evenly-distributed. Instead of having to rely on the judgment of a handful of people who are largely isolated from mainstream society on what is an efficient allocation of invested resources, it seems much safer and more stable to allow everyone to weigh in on the decision as to how much should be spent on everything.

This seems to me to be the classical argument as to why democratic governments are more stable in the long run than authoritarian governments - with more people giving their input, rulers can better-assess what the greatest needs of a society are and how many resources should be spent on each need. The only conceivable argument I can think of as to why it’d be better to have more wealth inequality in our society is if we were facing a cataclysm that required the resources of our entire country to combat - say, a problem like global warming. Sadly, I don’t see this argument being made by the wealthy elite a whole lot!

Articles for the day

Romney’s former partner at Bain makes the case for inequality - Discussed this up above. It’s a 6-page article, so be prepared for some reading.

A JP Morgan analyst makes the case that the Eurozone is improbable - The comparison to “countries beginning with the letter ‘M’” is priceless.

Atul Gawande details the past 200 years of surgery - It’s pretty incredible how far we’ve come in such a short period of time. The 300% mortality figure is pretty hilarious.

Things I did today

Today I woke up pretty late, so I wasn’t terribly productive. Tomorrow will be better!

Called AT&T and got a refund on spam subscription charges from MobiBroIQ back during ACen
Checked in with parents and talked about a possible longer-term job opportunity at a startup (this is really exciting and hopefully will pan out). Did some research on the company.
Printed out and filled out employment forms for the “flypaste” job - will turn these in tomorrow
Styled this blog to where I’m kind of comfortable with it. I wish I was better at design - I really want to find a good way to keep the black background and lighter-coloured text, but for longer reads it looks too harsh.
I rewatched Kannagi (which I’d bought at ACen last weekend)! The video quality wasn’t the best, and there were a few moments where the artists were probably drunk, but overall the series retained its charm really well. I’m really pleased with my purchase and am really happy that my original impression of the show has held true.

Hello World!

Thu, 03 May 2012 04:20:56 -0800

So this blog has been dead for the longest time! I’ve just now gotten to reviving it and have resolved to make one post every day from now on, no matter what it’s about. I’ll try to stick to the stuff that’s in my Google Reader and side projects I’m working on, so that’ll primarily be US policy and Rails/Python development.

Anyways, just wanted to touch base with y’all. The site will be pretty ugly for awhile (I think it’s sort-of passable now) so please bear with me, and please look forward to the posts from now on!

China 2011 Trip - Day Eight

Sun, 03 Jul 2011 12:06:21 -0800

Today (well, technically yesterday) was the start of our trip to Jiuzhaigou and Chongqing. We flew down at ~10am and arrived after a three-hour flight. It was 38C outside, and today was considered a cool day by Chongqing standards.

After checking out our hotel rooms, we rested for a bit in the heat until another family arrived - I think the kid’s name was Thomas? - and then we went to have dinner, which turned out to be the spiciest hot pot I have ever had in my life.

Then we drove up to a vantage point overlooking the city center and took photos. It was pretty cool - though vertigo-inducing - to see the whole city at once laid out pretty majestically. Then we came back home at around 11pm and passed out.

Thoughts:

And I thought Beijing was hot. This place is sweltering. Gonna be fun running around in 40C+ heat sweating like crazy.
Beijing International Airport is pretty cool. Not just literally. When you walk in the ceiling slopes up and away from you, giving you the sense that this place is on a scale of its own.
Apparently the normal Chinese reaction to eating beyond your spiciness tolerances is to get a massive stomachache. I can’t honestly say that that’s happened to me before.
Chongqing is pretty huge. Population-wise obviously, but the city center may be larger and taller than Chicago. It’s pretty cook to go to a city that basically nobody outside of Chinese people are aware about and realise that it’s bigger than anything you’ve ever experienced before by an order of magnitude.

China 2011 Trip - Days Five Through Seven

Sat, 02 Jul 2011 12:05:18 -0800

Days five and six consisted of us picking up one of my cousins (on my father’s side of the family) from the train stop and venturing around the Olympic grounds with her. My father’s been at the bi-annual China Heart Rhythm Society conference near the grounds. We had hotpot for dinner on the fifth day, and went to the Water Cube’s indoor waterpark for a couple of hours on the sixth. Then we visited the Bird’s Nest after meeting up with my father and drove by Tiananmen Square for the 90th anniversary celebration of the Chinese Communist Party. The entire surrounding area was lit up to celebrate, which was pretty cool.

Today my cousin was hanging out with one of my uncles, so my sister and I went ice-skating and practiced piano for a few hours in a shopping mall. Then we came back home and started packing for our flight to Jiuzaigou tomorrow morning. Not the most exciting of days to be sure, but it was fun.

Thoughts:

The Olympic grounds are surprisingly well-traveled long after the event has passed. I wish I could say the same of the Bird’s Nest specifically.
The propaganda surrounding the 90th anniversary stuff has been surprisingly low-key. There is the occasional garden sculpture around town or the ever-persistent tiny banner on all CCTV stations, but besides that I really haven’t seen much.
Some of the advertisements for the anniversary have been pretty revealing, though.
Beijing’s subway system is modeled almost exactly like Tokyo’s. The stations themselves, not the layout of the lines. I’m not sure which way the direction of causality flows.
Apparently Newegg exists in China. Consider my previous remark about someone getting very rich in this market redacted.
Designer apparel is ridiculously expensive in China. Even moreso than housing.
Communicating with people when you have a partial grasp of the language is really hard. Especially so if you’re trying to make a rigorous logical argument.

I’ve started taking photos of our trip, so I know that at least one person in particular will find some of the upcoming content interesting. Unfortunately our upload speeds at home are absolutely abysmal, so the photos will have to wait before being posted.

China 2011 Trip - Day Four

Wed, 29 Jun 2011 12:05:43 -0800

Today was mostly a logistics and planning day. Nailed down our trip down south, went out to eat with a couple of family friends at a Szechuan restaurant. Lots of free time. Ironically enough I almost forgot to write up this post - it is now 11:30 pm. So I’m half conscious now; you’ll forgive me if this post is a little more rambling and less coherent than my usual writing.

Things I thought about today:

Parents are people too, despite how noble or blameless they may want you to think they are.
Real life is incredibly dirty. Sweat and dirt and noise abound. I like it that way. Lots of people don’t.
I can’t imagine that the incredible amounts of air pollution going on in and around Beijing isn’t having a terribly detrimental effect on its inhabitants’ quality of life. I’ve gotta do some research in this area.
The ritual of returning to China every few years is just that - a ritual. There are parts to be enjoyed on my end, of course, but in the end it is for a higher purpose, of sorts.

China 2011 Trip - Day Three

Tue, 28 Jun 2011 12:04:19 -0800

Today we went to take some professional photos as a family.

It took eleven hours.

Thoughts:

Being a professional model must be incredibly hard. Christ.
The line between modernized and “old style” Beijing - red walls, sloped ridged green roofs, the like - is just that, a line that you can draw. It’s pretty jarring.
I feel like the “blue sky” record that Beijing recently was proclaimed as having broken must have had a pretty loose definition of a “blue sky”.
Once again, I am reminded of how easily your social norms can be warped beyond recognition without your realising it.

China 2011 Trip - Days One and Two

Mon, 27 Jun 2011 12:03:22 -0800

I’m in Beijing, China at the moment, visiting relatives for about two weeks. One of the things that’s struck me is how much the city has changed from just four years ago. I’ll be writing up a couple of longer posts fleshing out my thoughts on the changes after the trip; in the meanwhile, I’ll try to post some things that stuck with me at the end of every day.

Today’s musings:

Housing prices in China (Beijing, at the very least) are ridiculously high. I cannot begin to describe how high they are everywhere.
So is water. Water is like milk here in terms of price. Good for China.
Jackie Chan is also everywhere. If you ever wondered what he does in his spare time, it’s spent doing commercials. For everything.
I am consistently surprised at how good at Photoshop people can be when they’re forging brands and their associated styles.
The abundance of cheap labor in China means that we see some predictably menial service jobs available (elevator service in particular) that extract a pretty terrible toll on the workers over a long span of time.
Someone is going to get really, really rich bringing the electronic delivery system that currently exists in the US to China.

Correlations Without Correlata

Wed, 15 Jun 2011 12:03:14 -0800

I recently came across an interesting paper on quantum information theory outlining exactly why superluminal communications via quantum entanglement - so-called “spooky action-at-a-distance” - is impossible. If you’ve got some time and an interest in popular science I highly recommend it.

In the paper, the author makes reference to the idea that physical reality is merely illusion and that the only truly “real” thing about our existence is the communication and transfer of information between entities. The idea that we’re merely informational constructs in a higher-level simulation of sorts begs the question: who exactly is running the simulation? And for what purpose?

If you’re like me and don’t work with quantum information theory on a daily basis, this may come across as incredibly unintuitive and rather baseless; it sounds like something that you’d hear from an adolescent’s aimless musings, not from some of the most prominent physicists of our time. It may surprise you that this view of reality is the dominant one accepted by most theoretical physicists. This paper in particular has been passed around for some time already; the reaction from physicists tends to be surprise that anyone finds the ideas contained within the paper a surprise. Within the QIT community, these ideas are already considered obvious. All of this came as a pretty huge surprise to me.Aside from being blown away by the explanation of some of the quantum-mechanical phenomena that I’ve been hearing about for so long (with significant amounts of misinformation, it seems), the paper really got me thinking about the idea of correlations without correlata in modern society. I recently finished a re-watch of Ghost in the Shell: Stand Alone Complex, which talked in depth about this very concept. In GitS, the idea is addressed from a sociological standpoint; the series asks the question “how can societies and law enforcement in particular adapt to a futuristic society where the sheer complexity and volume of digital networks makes it impossible to seek out a social phenomenon’s source?” Interestingly enough, GitS doesn’t provide a definitive answer; you see Section 9, the law enforcement agency that the protagonists belong to, struggle to deal with a series of hysteria-provoked crimes where they’re not even sure that the originating incident had a physical perpetrator (hence the subtitle Stand Alone Complex). In the end, they have to rely on mistakes consciously committed by the originator in the physical world to track him down, and even then they barely manage to pull the case together.

The reason all of this is relevant is that we’re fast approaching the time where the idea of correlations without correlata and the “Stand Alone Complex” become very real. Already huge portions of the net are simply untraceable and unmonitored; in addition, every day we’re losing huge portions of our digital and cultural history to negligence. Just look at the efforts made by Google to save the early history of newsgroups; if one person had not made that seemingly-irrational decision to archive everything he could, we would have lost immense portions of the history of some of the first complex networks to exist on the net.

Some people out there might wonder why this is such a big deal, and why preserving as much as we can of our history is so necessary. The reason should be obvious - the very basis of our knowledge today rests on our ability to learn from mistakes we made in the past. That’s why we don’t use leeches in medicine anymore. That’s why we earthquake-proof homes in California and Japan (or why we know how to do this). We’re also incredibly bad judges of what will happen in the future; look at what people thought 2010 might look like in 1990, and you start to get a sense of this. Who’s to know what will be important for us to know or remember in ten years. Twenty? Our very survival as a race depends on our ability to remember what we’ve been through before, and with the advent of digital communications, we’re losing our ability to do just that.

The Difference Between Politicians and the People

Sun, 21 Nov 2010 12:02:06 -0800

One of the most frustrating things about discussing US politics is the disconnect between politicians and “normal people” - whether or not the representatives who are, in theory, supposed to defend the interests of their local constituencies are in fact interested in defending or even aware of these interests. It seems to me that this is one of those questions where everybody assumes their own opinion as fact without further proof - all too often I see people start or end their arguments with some variant of “well, politicians are corrupt”, or as in David Frum’s recent article discussing how the Democrats “got the better end of the deal” with respect to healthcare reform, that our representatives are better-served by putting into place policy that serves their constituencies.

As is, we’re getting a bad trade: Republicans may gain political benefit, but Democrats get the policy. In this exchange, it is the Democrats who gain the better end of the deal. Congressional majorities come and go. Entitlement programs last forever.

I’m not thoroughly convinced that’s true. It’s a pretty well-known fact that, when it comes to major entitlement programs and economic reforms, there exists a significant delay between when said programs and reforms are passed and when the general public starts becoming aware of the benefits of said programs and reforms. US politics exhibits strong hysteresis, if you will. Evidence for this sort of phenomena is everywhere - take one look at how people credit US presidents for economic booms and busts and you should be persuaded. As such, it doesn’t seem at all clear to me that the Democrats “got the better end of the stick” at all. It seems perfectly plausible to me that, five to ten years down the road, the American public will have forgotten the long, arduous journey that healthcare reform passage was for the Democrats, and the public consensus will be that healthcare reform is a product of both parties, passed by the Democrats but trimmed and shaped by the Republicans. And the terrible thing about this is that it makes passing legislation all that much more politically untenable; if we are to expect our representatives to put in the hard work of passing major legislation, the very least we can do to encourage this sort of behavior is to credit their work appropriately.

All in all I don’t think that this sort of thing is inevitable; the American public’s disinterest in correctly attributing blame and credit for political action seems like a solvable problem. Given a more open, well-behaved Congress I could see the American public being more interested in and well-educated about American politics, but it’s not going to be solved in a day to be sure.

More Social Norms Stuff

Wed, 14 Jan 2009 12:01:47 -0800

Via Matt Yglesias, a new USA Today/Gallup poll finds that the richer you are, the more likely you are to think that the economy is doing great:

The wording is pretty suspect on this one, I think; there’s still a huge output gap in the American economy that won’t be closed for years or decades as a result of the recent recession, but the economy is technically growing. And thank god for that. As a result, I’m not entirely sure how valid the findings in this particular poll are.

Regardless, these findings are pretty much in-line with what I said in August of ‘10: our most persistent indications of what the world is like comes from the people and establishments in our everyday lives, and so people in general cannot help but rely on that information when asked how they think the nation, or the world as a whole is doing. We all fall victim to this every day, and it is imperative that we force ourselves to recognize just how unrepresentative of reality that information truly is.

On a side note, apologies for not posting at all in the past few months. I’ve been hesitant to do so since I haven’t really had the time to commit to longer, more analytical posts; I think I’ve just decided to go with somewhat shorter, more frequent posts.

In the past few weeks, I’ve been working on a side project called LL Animu Recommendations, mostly. Been learning Ruby on Rails and re-learning algorithms from the ground up at the same time. It’s an attempt at making anime recommendations simple and fast by using a Danbooru-style tag system and user voting to filter out what you’re likely to enjoy. If you’re interested, give it a look! The code is up on Github, as well.

New Sheet to the Same Heights

Tue, 25 Nov 2008 12:01:11 -0800

Er, once in awhile I run into a piece I hear that’s not designed for piano solo. So I arrange it for the solo piano (usually by cutting the heart and soul out of the piece).

One such example I just finished with. “To The Same Heights”, or “同じ高みへ”, from the anime / visual novel Clannad, was originally designed as a piano duet. I spent a day slaving away on Sibelius to bring you this (admittedly awkward) arrangement for the solo piano. If anyone knows how to make the left hand less awkward please, for the love of God, tell me.

Here is the score in PDF format. And here’s the MIDI file for previewing.

Enjoy! And please, comment on what I can improve. I only dabble in arranging, so if anything needs fixing please let me know!

The Plasticity of Social Norms

Tue, 25 Nov 2008 12:00:39 -0800

I recently had a conversation with my father about income disparity and social mobility in the US. He’s always been a huge believer in social mobility in the US - you know, the belief that if you work hard, you’ll be rewarded for your efforts. I showed him recent data that suggests the contrary - that the US is actually one of the least socially-mobile nations in the developed world, and he was legitimately surprised at the findings.

I pointed out that this really isn’t a huge surprise, given the fact that income disparity in the US is so high - that the rich keep getting richer and their tax rates keep going down, while the poor stay more or less where they are. I pointed him to an Ezra Klein article talking about this and he was particularly intrigued by the “top 1% of earners in America accounted for 23.5% of the total income in the US” figure. Upon further inspection, we came across the source of the figure and my dad was absolutely shocked to learn that he was in the top 1% - but what shocked him more was just how low the bars are for the top 1%, top 5%, and top 10% (they’re $398,900, $155,400, and $109,600 respectively). He seemed completely unable to believe that the average American was earning so little that if you made more than $110k a year you’d be in the top 10%.

It was really fascinating to see this happen - to see someone suddenly realise that their normal, everyday experience is actually completely abnormal - and for the shock of this realisation to be so huge that the person just simply refused to believe the truth. Since my dad mostly talks with other physicians, his normal everyday experience is to see people who are like him - making enormous amounts of money every year, that is - so to an outside observer it’d seem natural for my dad’s definition of “normal life” to change from a first-generation immigrant enrolled in medical school and forced to literally moonlight as a dishwasher to pay the bills to a successful physician in the top 1% of all American incomes.

But for my dad, it was shocking. I suppose it’s even more surprising given his history - he came to the US in 1988 with literally nothing more than $400 in his wallet and a medical degree from a university in China that nobody in the US would recognize. He had to work his ass off to start over from the beginning, and he very vividly remembers the hardships he had to endure to get to where he is now. So it’s interesting to see that his perception of normality in the US has changed so drastically when his memories haven’t. And it’s really enlightening to see the realization happen firsthand, to begin to understand why the people leading the firms on Wall Street, people who have been doggedly working in the financial system for their entire lives and have become successful, can begin to think that they deserve their incomes, or that their experiences are ordinary and that they’re normal people. It’s a bit humbling, in a way, to see that human memory is such a fragile construct.

New Project

Tue, 25 Nov 2008 11:58:00 -0800

I’ve started a new project, entitled MultiGrid. It’s a 2D shooter written in C++/OpenGL. I’m writing it from scratch, and will hopefully implement internet multiplayer capabilities. Check it out!

Spring Break

Tue, 25 Nov 2008 11:57:06 -0800

… and I’m spending it at home, sleeping the days away. How wonderful is life?

So it turns out that my return transportation was arranged under the assumption that my spring break was two weeks instead of one, which is really puzzling. So now I’m getting back to O’Hare at 10PM on Sunday night, and there’s about a two-hour bus and train ride back from there, so… I’ll be getting back late, yeah. Eh, at least I get to come back on time, right?

I’ve been doing a lot of piano playing, a little studying for Chinese, a little learning Brainfuck, Java, and the OpenGL interface for C++, and a tiny bit of studying chem for our pre-chem 12300 exam when we get back. I don’t get why we’re taking another cumulative exam right after we took the final, but whatever. I guess they make the rules for a reason.

Also, Clannad is almost done. The storyline is finished; the last episode will be a DVD extra that wraps up the series. It’s a good series, though the ending wasn’t nearly as good as that of, say, Haruhi or Kanon. But those are pretty high standards to be holding any anime up to. In any case, now that it’s done, they can get back to releasing new eps of Haruhi (or so I’ve heard), so this is a joyous occasion.

I’ve become a massive weeaboo, in case you couldn’t tell. Just to further prove my point, here’s me learning the Kanon intro (Last Regrets by Key) and the Haruhi intro (Bouken Desho Desho? by Aya Hirano).

Oh, and I think Joe and I are going to be writing an FPS. A fractal FPS. In Brainfuck.

God help us.

End of Winter Quarter Update

Tue, 25 Nov 2008 11:56:21 -0800

So we’re nearing the end of winter quarter. This week is finals, which means no classes but mass tests, and next week is spring break. Officially, winter quarter ends this Friday.

It’s been a long road. Winter quarter feels the longest at UChicago, for several reasons - the first being that it is the longest, technically. The weather is also absolutely terrible compared to the other two quarters. This is objective, by the way - no matter how much you enjoy winter weather, having to cross the Midway to and from the dining hall is a trial I wouldn’t put anyone through. Also, the professors figure that you’ve had a quarter to settle back into the groove of academics, so they ramp up the difficulty and speed of the quarter, as well. All in all, it’s certainly an ordeal that is one-of-a-kind. I’m just really, really glad I’m getting out of here for a week.

Registered for spring quarter classes today, too. Math 1990 Intro to Analysis and Linear Algebra, Chem 12300 Honors General Chemistry, Huma 12500 Human Being and Citizen, and Chin 11300 First-year Chinese for Bilingual Speakers. The schedule is essentially the same, but I’m really looking forward to spring quarter, for several reasons - first, I’ll actually be taking an actual math course. Second, one of my housemates is transferring chem lab sections into my section, so I’ll have someone to wake me up in time for lab on those days where I sleep in. Three, it’s spring quarter. The reawakening of the earth from its annual hibernation. Christ, I’m looking forward to it so much.

Finals week for winter quarter. This week, I have four finals - I turned in a final paper revision for Hum today at 12PM, I have a Chinese final on Wednesday at 1030AM, I have a chemistry final on Friday at 830AM, and I have a math final on Friday at 1130AM. So tomorrow (today) and Thursday are my cram days, I guess. Wish me luck. After all that’s done, I have a flight to MN at 8PM on Friday, so I won’t get much of a chance to take a breather and hang out with friends.

See you all in a week when this is over, then.

Github on and some change

LLM benchmarks like SWE-bench are not trustworthy

Predictions for 2025

US

National

NYC

Housing

Congestion pricing

Second Avenue Subway

California

Housing

HSR

Rest of world

Canada

Ukraine

China

Argentina

Tech

AI

Other

Pop culture

Sports

Following up on my 2024 predictions

Last year’s predictions

US

Congestion pricing

Second Avenue Subway

California HSR

Other

Rest of world

Tech

How’d I do?

Notes from Midtown East Casino Proposal Town Hall, 2024/01/11

Predictions for 2024

US

Rest of world

Tech

The US government is 13x better at healthcare insurance than private companies

Debugging stories: the inconsistent database

Setting this blog up on Github Pages

What’s Github Pages?

How do things work now?

How’s this set up?

Random musings

Catching Us Up to 2018

Catching Us Up to 2017

And We're Back!

Our Summer Family Trip to Europe - Prelude

On the Supposed Conflict Between Individual Rights and Communities

What's Been Going On

Forecasting Using Seasonal Adjustment Factors

Spring Break 2013 in Chile

Day 1 - Santiago

Bit of an update on Animurecs

ECLS-K: A General Overview

Detecting Subcommunities Within Social Networks

Anything Can Be Made to Sound Silly

Budgets and Rational Allocation

Links

A Brief Status Update on Where I'm At

Links

Treating Your Opponents Like Human Beings

More Links! I'll Be Writing at Length Tomorrow

Links! Links for Everyone!

Startups and the Homeless

Links

Better Links Than Never

Overconfidence and Communication

Links

Kicking the Daily Blogging Off!

Articles for the day

Things I did today

Hello World!

China 2011 Trip - Day Eight

China 2011 Trip - Days Five Through Seven

China 2011 Trip - Day Four

China 2011 Trip - Day Three

China 2011 Trip - Days One and Two

Correlations Without Correlata

The Difference Between Politicians and the People