It reminded me about how every time I land back in Europe or in Asia, I can immediately tell which apps have a global CDN or edge computing. Everything just immediately feels a little slower.

There’s some debate about when users start feeling latency, but it’s widely accepted that increased latency will impact conversions and user behavior significantly.

The figure often thrown around is that for every 100ms of latency amazon lost ~1% of sales. While I don’t know how true this is (especially given how slow the amazon app feels), it is something I know first hand. While at Pinterest we spent a ton of time improving the performance of our landing pages, as the faster it was the more users would convert and the more content they would consume.

Floored Latency

The speed of the experience you can offer your users is floored by a few variables, but most notably the physical distance from the origin server to where the user is actually sitting.

Us-east-1 (appropriately located right by “Centreville, Virginia”) is one of the main data centers run by AWS. If you’re a startup (or even quite a few mature companies), you are likely to deploy your application here. If you manage a stateful service, or your architecture doesn’t support distributed compute, you are likely to only deploy here. If you have users in Sydney, Australia, any network request you make will need to travel 15,677km to make it there, as the crow flies.

Distance from Centreville, Virginia to Sydney, Australia

If you’re traveling at:

1) The speed of light

2) as the crow flies

3) with no overhead

then that means that you are floored at 104ms of latency for your request.

Time in takes for the speed of light to travel from us-east-1 to Sydney

And this is assuming no interference, other traffic, or time spent handling the request.

In reality, the ping you’ll experience will be worse, at around 215ms (which is a pretty amazing feat in and of itself - all those factors above only double the time it takes to get from Sydney to the eastern US).

Ping to various cities around the globe

Realized latency

Having spent so much time trying to optimize web pages and API responses for performance, I’ve gotten a pretty good internal model for latency. I can’t quite tell the difference between a us-east-1 server when I’m in New York versus San Francisco, but I can definitely tell if you’ve got an instance deployed in eu-central-1 or not when I’m in Italy, or ap-east-1 when I’m in Sydney.

Using a global CDN can help get your assets to your users quicker, and most companies by this point are using something like Cloudflare or Vercel, but many still only serve static or cached content this way. Very frequently the origin server will still be a centralized monolith deployed in only one location, or there will only be a single database cluster.

As soon as you land back in the United States and turn of Airplane mode on your phone everything just starts feeling… snappier? A little more fluid? As much as T-Mobile and Verizon would like to take credit for that I don’t think theres much more to it than the physical location of the servers and where you are at that moment.

I wanted to explore how it worked, and what it was actually doing under the hood.

Web3 Wallets

There are broadly two options for storing your crypto assets - custodial and non-custodial.

A custodial solution is something like coinbase - typically a single company, which manages the private key to your wallet for you, and which is the centralized gatekeeper to your funds.

A non custodial solution is one in which you store the private key yourself - the wallet simply provides the software that you run locally that generates and stores the key. This looks like Metamask or Phantom - you will typically install a chrome extension, go through an onboarding flow, save your recovery pass phrase, and only then can you begin to use it on various dapps or to move tokens around.

Generally speaking, the user experience with custodial services is better, and allows for easier user onboarding. However, it comes with some fairly major downsides, including centralization and greater likelihood of regulatory scrutiny.

Magic is somewhere in between - they say they are non custodial, but offer the UX of a custodial solution. No need for a user to install a separater chrome extension - they just need an email address and their browser.

Under the hood

Magic is relies on AWS for their product. Their sign in and user logic uses AWS Cognito, and their key storage uses KMS.

AWS Cognito is the identity solution offered by AWS - they will handle authenticating your users for you, and offer a wide variety of sign on methods, including email/password, phone number, 3rd party, and magic email.

When setting up a Magic wallet, you start with your email address. This will talk to AWS, which will send a link to your email address containing a unique token.

Embedded Magic Link form (in this case, ImmutableX)

Confirmation code for auth sign in

Once you’re on the verification page, your browser will talk to AWS to authenticate you, and Magic will start creating your key.

They will first generate the private key for the chain you’re using in the browser, using javascript, and keep that in memory. This is the root key material, which is used to sign transactions for whichver chain you’re using - this can be imported into metamask, a hardware wallet, etc.

This key, which we’ll call user key (UK), is never shown to the user. If you intercept the network request for initial encryption you’ll see it, base64 encoded.

It will then talk to KMS, using the user account that was just authenticated for you, and create a new key within KMS. This is the key that lives inside of Amazons data centers, and which will be used to encrypt and decrypt the UK.

Your browser will send the UK in plain text to KMS directly, never speaking to Magic links servers. It will retrieve the encrypted material, and store that both within the browser cache and with magic link.

Decrypted private key coming back from KMS

If you copy and paste the “Plaintext” from above into Metamask, you can use your Magic.link private key outside of the Magic ecosystem.

Once they have the encrypted key contents, the decrypted key in memory, and the public key, magic will create an account for you on their centralized servers.

{
  "data": {
    "auth_user_id": "my-magic-link-id",
    "auth_user_mfa_active": false,
    "auth_user_wallet_id": "my-wallet-id=",
    "challenge_message": null,
    "client_id": "my-client-id",
    "consent": {},
    "delegated_wallet_info": {
      "delegated_access_token": "{\"ciphertext\": \"cipher text\"}",
      "delegated_identity_id": "us-west-2:aws-key-id",
      "delegated_key_id": "delegated-key-id",
      "delegated_pool_id": "us-west-2:delegated-pool-id",
      "should_create_delegated_wallet": false
    },
    "encrypted_private_address": "long base64 encoded encrypted private key",
    "encrypted_seed_phrase": null,
    "hd_path": null,
    "login": {
      "identifiers": [],
      "oauth2": null,
      "type": "email_link",
      "webauthn": null
    },
    "public_address": "0xmypublickey",
    "recovery_factors": [],
    "utc_timestamp_ms": 1687790408203
  },
  "error_code": "",
  "message": "",
  "status": "ok"
}

Any subsequent logins follow this same setup, with the only difference being that once you’ve authenticated with AWS, your browser will first check with Magic to see if you already have an encrypted key stored with them, and if so, will retrieve the encrypted key and decrypt it using AWS KMS.

Is this safe?

The common refrain amongst crypto purists is “not your keys not your tokens” - this is in reference to custodial services, like Coinbase and FTX.

Magic is being a bit disingenous when they say they say they are non custodial - while it’s true that in their current set up they don’t directly have access to your raw private keys, they are still the admins of their AWS account. They are limited only by the policies they themselves have put in place - they can just go into their AWS console and change the key policy, and decrypt the encrypted private keys they have for every account.

Additionally, every time you authenticate on a new device, the raw key is transmitted over HTTPS to your machine. If your machine is being man in the middle’d, an attacker will be able to see your raw key materials if you sign in. Additionally, because this is in-browser, Magic is not using any form of certificate pinning for AWS.

This isn’t ideal. This isn’t MPC or any sort of advanced cryptography - it’s just using AWS as your trust layer, and some clever engineering to make the user experience good. I’m a bit skeptical of their claims that this isn’t custodial, since Magic technically has the ability to just recover everyone’s private keys. This might be good enough, though, and the users magic is targeting (web3 gamers and casual users) are’t likely to care about the security implications.

I’ve always been surprised at how slow and and clunky it is to search in iMessage. It feels like it doesn’t search through your whole history, the UI for the results is too small given the prominence of the feature, and there exist quite literally no ways to filter or refine your search.

I realized that iMessage just stores its database locally as a sqlite file, so I went about building an alternate UI for searching, and adding in a few features that I thought would be interesting. These include:

• Semantic Search - I wanted to create embeddings for every message/conversation and then add proper semantic search on top of it

• Wrapped - I really like seeing stats about my life, and I really enjoy what Spotify has done with Wrapped, so I set out to do the same for iMessage

• AI Conversations - Talk with your friends, AI-ifiied; once you have all the context around a conversation, you can understand how that person texts and what their tone it. Then it’s as simple as plugging it into ChatGPT

• Export - Export your conversations in JSON or plain text

Your iMessage Wrapped

Reverse Engineering the Database Schema

The database sits at /Library/Messages/chat.db, while all the attachments live in ~/Library/Messages/Attachments. I made a copy of it and then opened it up and datagrip and then was ready to go. The database schema is relatively straightforward to understand - there where message, chat, handle, and attachment tables, each containing the data you’d expect. A rust library named imessage-exporter was particularly useful for understanding how the tables were joined together and what queries to make.

A few gotchas were:

• iMessage does not receive raw text as its messages - it actually receives a typedstream, and then later on post processes it and backfills in the text column. There’s a fairly obscure library for parsing these in typescript (with a bug fixed) here

• When using Messages in iCloud, your messages app won’t have all your conversations and attachments. This will make your old conversations look much more sparse than they really are. You can force iMessage to download all your conversations by toggling it on and off in iMessage settings.

Wrapped

Wrapped gives you a breakdown of your iMessage habits, broken down by year and by conversation/person.

I actually used ChatGPT to generate the product features included in iMessage Wrapped - as soon as it can do figma mocks I’ll redesign the UI using that, as well.

ChatGPT's suggestions for the features in wrapped

AI Conversations

I’ve always thought the amount of information stored in iMessage wasn’t being used appropriately - I thought that it could learn more about you, or the person you’re talking to, and suggest more things or be smarter about your interactions.

One of the first ideas that came to mind when I was building this was to use GPT4 to just continue any conversation, right where you left off.

AI generated chat that really can't make it to dinner

I started building Mimessage on April 1st, and just a few days ago I saw someone on hackernews had had the same idea to clone their friends. I think that training a model is actually a much smarter way of accomplishing this, and seems to lead to better results than naively continuing the conversation with GPT4. I’ll try and get it running locally with LLaMA soon.

Better Search

I added in global search with filters, as well as a chat specific search that allows you to do either fuzzy matching using fuse.js or just writing raw regex in the query itself.

Global search with filters

I also created a virtual FTS5 table in the sqlite messages copy, which does MATCh searches ordered by rank, and raw LIKE %QUERY% queries as well. This is really really fast, and a way better text searching experience than iMessage.

Semantic Search

I also wanted to add semantic search, as those results will often blow pure text searches out of the water. I used OpenAI and ChromaDB to create and store the embeddings for each text message, respectively.

Enabling semantic search on top of imessage

This takes quite a while and costs money, as well as requiring you to send your data off to OpenAI, so I’m planning on migrating to an entirely on device version soon.

Yak shaving the actually cool part

The coolest part of this project was actually in using a tool I created for this project called repo-refactor - it’s a library that will convert a github repo from one language to another https://github.com/jonluca/repo-refactor. The rust library I mentioned above had already done some of the heavy lifting in creating data models and structures representing the schema, as well as writing some of the base queries. This was really useful for creating and understanding types early on, and for actually reading the rust code (admittedly not my forte).

The tool is very much in its infancy, and works successfully somewhere between 40 and 80 percent of the time. It doesn’t do great on long files, and some clear failure modes (going from weakly typed -> strongly typed is pretty bad), but I think there are some cheap techniques that can be implemented to make it way more accurate. It’s still pretty incredible that it works this well with very little prompt engineering or manual cleanup.

Next Steps

The whole project was built in open source here and you can grab a copy of the latest working code from the releases page. I want to get the inference running entirely locally, probably with alpaca or with vicuna weights, so that it’s free and privacy preserving.

I also want to add more stats to the wrapped page, and make it more of an experience like Spotify’s is - if there’s any designer reading this that wants to collaborate on it, shoot me an email

Their UI was pretty bad, and it was quite slow. It also wouldn’t let you search without an area code, and the majority of area codes that I tried said that they didn’t have any numbers associated with them. I figured it would be faster to do this programatically.

Finding the endpoints

I used burpsuite to find out which API endpoint Verizon was hitting to fetch the available phone numbers.

From here I could just export it to code - my preferred method is to copy the request as curl from within BurpSuite and then use a tool like CurlConverter to turn it into the language of choice, although BurpSuite does have plugins that do this natively as well.

Calling the API programatically

Curl converter gave me a nice little code snippet that I could use. I put it into a jupyter notebook and was happy to see it all worked. Their endpoint didn’t seem to use any signing or on-device auth that would’ve made this difficult, besides a static basic auth token as an HTTP header.

import requests

headers = {
    'Host': 'api-v.vzmessages.com',
    'Cache-Control': 'no-cache',
    'Connection': 'close',
    'Accept': '*/*',
    'User-Agent': 'Verizon%20My%20Numbers/1 CFNetwork/1333.0.4 Darwin/21.5.0',
    'Accept-Language': 'en-US,en;q=0.9',
    'Authorization': 'Basic <Auth-Token>',
}

params = {
    'state': 'CA',
    'acode': '213',
}

response = requests.get('https://api-v.vzmessages.com/VirtualNumber/listOfMVNs/US', params=params, headers=headers, verify=False)

It seemed like multiple calls to this endpoint with the same params gave different replies, which meant it was non deterministic.

It also seemed like if you removed the params, it would remove the filter and return back many more numbers - I didn’t really care what the area code was, so removing the params should make it easier to find a number I liked.

I wrote a quick script to repeatedly call the API and try and find a number with at least 3 consecutive numbers.

def contains_cons(num):
    for i in range(len(num) - 2):
        if num[i] == num[i + 1] and num[i + 1] == num[i + 2]:
            return True
    return False

and then just kept calling that infinitely.

seen = set()
while True:
    response = requests.get('https://api-v.vzmessages.com/VirtualNumber/listOfMVNs/US', headers=headers)
    nums = response.json()
    for num in nums:
        if num in seen:
            continue
        seen.add(num)
        is_valid = contains_cons(num)
        if is_valid:
            print(num)
            break

Finding valid numbers

I let this run for a few minutes and checked back and found that there were quite a few good candidates.

These are the numbers that had at least three consecutive digits (along with the number I actually ended up using, blacked out).

Purchasing the number

I went through and bought a dummy number to figure out how to purchase the number programatically, and then did the same method as above to purchase it. Luckily there weren’t any unique IDs or anything, I could just swap out the number and it would work.

import requests

headers = {
    'Host': 'api-v.vzmessages.com',
    'User-Agent': 'Verizon%20My%20Numbers/1 CFNetwork/1333.0.4 Darwin/21.5.0',
    'Connection': 'close',
    'Accept': '*/*',
    'Accept-Language': 'en-US,en;q=0.9',
    'Cache-Control': 'no-cache',
}

json_data = {
    'countryCode': 'US',
    'zipcode': '<your billing zipcode',
    'deviceId': '<your device id>',
    'contentId': 'Content2',
    'price': '15',
    'mvn': '<number you want>',
    'mdn': '<your hard line number>',
}

response = requests.post('https://api-v.vzmessages.com/VirtualNumber/purchase', headers=headers, json=json_data)

Success

Using this I was able to purchase the number, and now it shows up in the app. One drawback is that it’s not really integrated fully in your phone - for instance, iMessages to that number won’t work, and the calls need to be made through the app. It’s still nice to have, and a nice number to give out when you don’t want to give out your real one.

Luckily Resy’s APIs were fairly straightforward - their web and mobile APIs were very similar, an didn’t seem to have any rate limiting.

Notifications

Resy already supports “priority notifications”, where you can sign up for notifications for specific restaurants.

However, there are a few issues with this.

1. You have to manually set these up for each date you’re interested in

2. These won’t automatically book them for you - if availability comes up when you’re not near a device to book it, you’ll probably miss out

3. I’m not sure how quickly they actually push these out. I think there’s some delay to allow their concierge team to nab reservations first

I wanted to build a full end to end flow which will make both notify you that a reservation is available and book it for you.

Clients

I started by checking out their front end code and trying to de-minify it.

Interestingly, the webpack config Resy is using isn’t mangling their code or removing comments, so it makes directly using their client fairly straightforward. You can rip code segments like these almost one to one.

getReservations(params) {
  return RequestBuilder.get({
    endpoint: "4/find",
    data: {
      lat: 0,
      long: 0,
      day: params.day,
      party_size: params.party_size,
      venue_id: params.venue_id,
      resy_token: params.resy_token,
    },
  }).then((response) => ResyTransformer.finder4(response));
},

There was also some classic hacky code, like specifically excluding some assets from the UI when they match a property ID.

function getServiceTypes(serviceTypes, venue, venueId) {
  // a default for a service type that is not an available service type for the venue
  const temp = [
    {
      available: [],
    },
  ];
  serviceTypes.forEach((service) => {
    // EXCEPTIONS - lol
    // Noble Experiment doesn't serve dinner.
    if (venueId === 569 && service.value === "dinner") {
      temp[temp.length - 1].serviceTypeName = "Cocktails";
    }
  });
  // Robert Sinskey Vineyards doesn't serve food either (885)
  // and one more time because europe The Mulwray UK (2040)
  if (venueId === 885 || venueId === 2040) {
    temp.forEach((item) => {
      item.serviceTypeName = "";
    });
  }
  return temp;
}

and some nice copy + paste

// shamelessly stolen from https://toddmotto.com/understanding-javascript-types-and-reliable-type-checking/
[
  "Array",
  "Object",
  "String",
  "Date",
  "RegExp",
  "Function",
  "Boolean",
  "Number",
  "Null",
  "Undefined",
  "Window",
].forEach((type) => {
  Utils["is".concat(type)] = (
    (self) => (elem) =>
      Object.prototype.toString.call(elem).slice(8, -1) === self
  )(type);
});

Their API is pretty full featured and clear, though, and understanding how their availability and searching worked was easy.

Hooking it all up

I wrote a few small services for wrapping their code and logging the user in to get their authentication token, as well as for sending a text message with twilio. This will refresh my auth every day, and fetch availability every 5 minutes.

const refreshAvailability = async () => {
  log.info("Finding reservations");

  await venuesService.init();
  const venuesToSearchFor = await venuesService.getWatchedVenues();
  // You get more availability if you have an amex card and you log in

  for (const venue of venuesToSearchFor) {
    await refreshAvailabilityForVenue(venue);
  }
  await venuesService.save();
  log.info("Finished finding reservations");
};

const regenerateHeaders = async () => {
  await service.generateHeadersAndLogin();
};
// every day fetch every post
cron.scheduleJob("*/5 * * * *", refreshAvailability);
cron.scheduleJob("1 * * * *", regenerateHeaders);

regenerateHeaders().then(() => {
  refreshAvailability();
});

And then the logic for actually understanding if the dates work

const refreshAvailabilityForVenue = async (venue: VenueToWatch) => {
  try {
    const availableDates = await service.getAvailableDatesForVenue(
      venue.id,
      venue.partySize
    );
    if (!availableDates.length) {
      return;
    }
    for (const dateToCheck of availableDates) {
      const slots = (await service.getAvailableTimesForVenueAndDate(
        venue.id,
        dateToCheck.date,
        venue.partySize
      )) as EnhancedSlot[];

      const possibleSlots = slots.filter((slot) => {
        const start = dayjs(slot.date.start);
        const minTime = dayjs(`${start.format("YYYY-MM-DD")} ${venue.minTime}`);
        const maxTime = dayjs(`${start.format("YYYY-MM-DD")} ${venue.maxTime}`);
        slot.start = start;
        return start >= minTime && start <= maxTime;
      });

      if (possibleSlots.length) {
        await parsePossibleSlots(venue, possibleSlots);
        return;
      }
    }
    log.debug(`Found no valid slots for ${venue.name}`);
  } catch (e) {
    console.error(e);
  }
};

Setting up the monitoring

I wanted some form of persistence so that if a restaurant released a lot of availability all at once my script wouldn’t rebook the same restaurant over and over. I used lowdb, a small JSON based “database” that stores your data in a file. I wrote a simple schema that you fill out, like what your time ranges are and what your preferred time is, and whether it should attempt to book it for you.

{
  "venues": [
    {
      "name": "Le Bernardin",
      "id": 1387,
      "notified": true,
      "minTime": "18:00",
      "preferredTime": "19:30",
      "maxTime": "22:00",
      "shouldBook": true,
      "uuid": "7088f322-578d-4b10-816f-44c4213118dd",
      "partySize": 2
    }
  ]
}

Success

A couple days later it worked, and I got a booking at Le Bernardin for my girlfriend and I.

Last one left to do now is Tock!

Code

The code is open source on GitHub here. It’s probably in a state that a software engineer could get it working with the right environment variables, but it’s not necessarily ready for anyone to be able to use it out of the box. PRs are welcome though!

This investigation uncovered similar vulnerabilities in NFT exchanges, yet to be publicized.

Background

Metaplex’s Candy Machine, a Solana program which handles the logistics of NFT issuance, just launched last September. You instantiate it with their CLI, feed it your images, and it handles the rest. It will deal with all the technically complex parts of putting the images on chain and creating the smart contracts to mint them to the buyers.

It’s extremely simple to launch an NFT sale with Metaplex; you choose the price you want to set, the timing of the collection drop and any other configs - it handles the rest and mints right to recipients wallets.

This simplicity greatly lowered the barrier to entry - you didn’t need to have any Rust knowledge or Solana API experience to use it. When it first came out it led to a huge increase in NFT collections.

Since its inception, over 14,800 candy machines have been created, each corresponding to an NFT collection.

Impact

The goal of this research was to identify how the attacker exploited the vulnerability, trace the funds and their total dollar denominated value, and then to determine which projects were impacted.

The attacker targeted 4,410 of the 14,800 candy machines that were created at the time. I’m guessing they didn’t target every vulnerable program because they had trouble pulling the historical candy machine creation records.

They fired off withdrawal transactions that took advantage of the reinitialization bug over the period of an hour.

The withdrawal transactions lasted between 5:57am and 6:49am EST on January 4th 2022. At 6:20am, the patched contract was deployed, causing every subsequent transaction by the attacker to fail.

Of the 4,410 candy machines targeted, 3,470 were completely drained. The vulnerability didn’t give the attacker permanent control of the candy machines - only for the duration of that transaction, which means that the candy machines that were impacted are not currently vulnerable.

Some of the notable projects impacted by this vulnerability are SolSteads, Contrastive, and Degen Ape Society, with a full list below.

Vulnerability

The bug was subtle - the attacker was injecting pre-initialized accounts and the program was not checking if the account had already been initialized, meaning an attacker could populate their own address as the authority of the contract.

The fix itself was fairly straightforward.

The hack seems fairly unsophisticated - the damage this vulnerability could do was pretty high, as the bug effectively allowed any account to control the Candy Machine. The attacker submitted the transactions slowly, and would probably have been able to capture the entirety of the vulnerable set of candy machines had they submitted the transactions through their own RPC pool without rate limits.

What’s also interesting about the fix is that it was actually fixed in code on December 31st for Candy Machine v2, but the CMv1 contract wasn’t redeployed until it was actively being exploited.

Fund extraction

The attacker used Serum DEX and RaydiumSwapV2 to convert the SOL to USDC, then sent the USDC to a FTX address. It should be fairly easy to reverse their idea from FTXs end if they’ve KYC’d properly.

Candy Machine

Candy Machine v1 is now deprecated, and any new candy machines created should be v2s. From their docs:

The second iteration of the well-known Candy Machine, a fully on-chain generative NFT distribution program, provides many improvements over its predecessor. The new version also allows you to create a whole new set of distribution scenarios and offers protection from bot attacks, while providing the same easy-to-use experience.

Research

Querying for historical data on chain in Solana is a time consuming process. I tried doing research in jupyter notebook at first, but the volume of data made it hard to parse and query.

I ended up cloning the historical transactions into a local database, and indexing that for faster queries.

export class MongoClient {
  init = async () => {
    log.info("Connecting...");
    await connect("mongodb://127.0.0.1:27017", {
      keepAlive: true,
      keepAliveInitialDelay: 300000,
      dbName: "candymachine",
      minPoolSize: 50,
      maxPoolSize: 500,
    });
    log.info("Connected to mongo db");
  };

  saveHashes = async (hashes: object[]) => {
    log.debug("Saving batch...");
    try {
      await Txhashes.insertMany(hashes, { ordered: false });
    } catch (e: any) {
      // ignore dup key errors
      if (!e.message.includes("E11000")) {
        log.error(e);
      }
    }
    const count = await Txhashes.count();
    log.debug(`Saved batch - ${count} total documents`);
  };

  getHashes = async (filter: FilterQuery<typeof Txhashes> = {}) => {
    const docs = await Txhashes.find(filter).limit(25000);
    return docs;
  };
}

I first cloned all the transaction hashes into Mongo - I set up a connection pool of various RPCs to accomplish this, as there’s no way of getting it from the Solana mainnet-beta RPC in a reasonable amount of time.

const history = await con.getSignaturesForAddress(
  new PublicKey(publicKey),
  options
);

for (const c of chunk(history, 100000)) {
  await mc.saveHashes(c);
  log.info("Completed chunk");
}

Then, after fetching all the hashes, I would clone the parsed transaction details into Mongo

const run = async () => {
  await mc.init();
  log.info("Fetching hashes");
  let hashesToFetch = await mc.getHashes({ tx: null });
  log.info("Fetched hashes");
  let i = 0;
  while (hashesToFetch.length) {
    let isNearEnd = hashesToFetch.length < 10000;
    const chunkSize = isNearEnd ? 10 : 200;
    if (isNearEnd) {
      shuffle(hashesToFetch);
    }
    const chunkedHistory = chunk(hashesToFetch, chunkSize) as string[][];
    const processChunk = async (hashes: any[]) => {
      let message = `Fetched txs ${i}`;
      let savedTxMessage = `Saved txs ${i}`;
      i++;

      console.time(message);
      const hashMap = {};
      hashes.forEach((hash) => {
        if (hash.signature) {
          hashMap[hash.signature] = hash;
        } else {
          console.error("what");
        }
      });

      try {
        const { c, tx: txs } = await fetchTxsWithFallbackWithConnection(
          hashes.map((p) => p.signature)
        );

        console.timeLog(message, c?._rpcEndpoint);
        console.timeEnd(message);
        let failureCount = 0;
        txs.forEach((tx) => {
          if (!tx) {
            failureCount++;

            return;
          }
          tx.transaction.signatures.forEach((s) => {
            if (hashMap[s]) {
              hashMap[s].tx = tx;
            }
          });
        });
        if (failureCount) {
          console.error(`Invalid txs: ${failureCount}`);
        }
        console.time(savedTxMessage);
        await Txhashes.bulkSave(hashes);
        console.timeEnd(savedTxMessage);
      } catch (e) {
        return;
      }
    };
    try {
      const promises = chunkedHistory.map((h) => limit(() => processChunk(h)));
      await Promise.all(promises);
    } catch (e) {
      console.error(e);
      console.error(
        `Chunk failed with total history length of ${hashesToFetch.length}`
      );
    }
    console.log("Finished chunk");
    log.info("Fetching hashes");
    hashesToFetch = await mc.getHashes({ tx: null });
    log.info("Fetched hashes");
  }
};

I also pulled every transaction (legitimate and the attackers) that called the withdraw function on the candy machines.

Of the 14,800 candy machines, 11,848 have had the withdraw function executed on them. The top accounts associated with these functions are below.

Only cHfYkrVAwfEoe3Mr2GbvzpNQJboDL6AiBoFZDsf8dxj seems to be doing this maliciously - the other accounts are all calling legitimate withdraw functions.

F9fER1Cb8hmjapWGZDukzcEYshAUDbSFpbXkj9QuBaQj actually seems to have created over 2,000 candy machines, and then attempted to call withdraw on them, single handedly creating ~14% of all candy machines on Solana.

[Redacted Pending Vulnerability Disclosure]

[Redacted pending vulnerability disclosure of Solana exchange]

[Redacted Pending Vulnerability Disclosure]

[Redacted pending vulnerability disclosure of Solana exchange]

Identified Projects

I went through and fetched the public keys to all known affected projects and tried to map them back to the hacked machines. I identified 334 unique projects that were actively listed on Magic Eden, Solanart, DigitalEyez, and Solsea as being affected.

012Funksaicy 0xDRIP 100Radials 169 Pixel Gang 3D Flowers A Pixel Art Abstergo Aeterna Civitas Afrobubble AI: Baby Bots Aircrafts AIRDOGS Alphabet Originals Angomon Angry BaboonS Angry Bunny Club Angry Citizens AngryWorms Anti Artist Club ApexDucks Halloween Arcade ‘88 Art by NRG – Pop Sushi Artificial Irrelevants Autistic Reindeer Herd Baby Ape Social Club Baby Bait Baby Frogs Baby Goblin Bad Bromatoes BADBOYS BalisariNFT Bannurs Bare Bones Society: The Kingdom Of Secrets Beat Drops V1 Boat Boys BoldFrames Boopie Gen 1 Bored Ape Social Club Boss Babes Broken Robot Burger Bar bugbearz Bunny Warriors CASSETS Audio CatPunk CATPUNK OG PASS Cats Club NFT Chickenz ChihuahuaSol Classic Art Mashups Coherence Combat Women NFT Contrastive Crazy Pickles Creepy Girls Crypto Greeks Crypto Idolz - Faces CryptoCream CryptoCubs CryptoCubs Mutants Cryptone™ Cryptonic Creations CryptoRock CryptoTeds CryptZero Season 2 - Ghosts Cult of Meerkats CyberKeys Danger Valley Danuki Dojo Deadass Deeps Degen Ape Society DegenEggs Degeneggs - Gen 2 Desolates Metaverse Dessert Girls DigiLife Dinos Zone Dippy Dragons NFT Doll Society Dragon Eggs NFT Dragon Slayerz Dreamland Monkeys Element Art NFT 2D Enigma Expanses enviro Enviro Epoch Labs Ethereans Eyeballz Eyes WTF Fallen Traveler Fast Food Thugbirdz Finefolk Floppy Disk Nft Flutter Fractures G.O.A.T. Collection Galactic Goose GAMEKIDZ GANder g0 Gem Heroes Ghostface Ghostly Sols Ghoulie Gang Gremlins NFT Hallow Birds Happy Pups Hellish Party Boollons Hello World! NFT Hemp Heroes and Villains: PASS High Roller Hippo Clique HOAG play Honorary Space Bums Hot Bunnies NFT Houses Of pixel Iceland - 0xDRIP Icy Bearz NFT Idle Planets - Autumn 2021 Moon Idle Planets - Holiday 2021 Moon Infamous Apes InnerMind Intersolar iTrading_Bot Jelly Beasts Jingle Monkeys Joeian’s Collection JOSEPHTAYLOR.ART: CRYSTAL BEAMS Jungle Cats Just Blocks Kitten Coup Knightdom KoreanPunkz Krunk Roach NFT Kyoudai Academy: Solana Arcade Games Labyrinth Lanabots Lazy Heroes Little Noots Los Cactus Hermanos LuchaLucha NFT Lucky Kittens LuxAI Mad Vikings Arms Collection Magic Solana Shits Make your own NFT Mark McKenna’s Heroes & Villains: Origins Megumi Meta Homes Metadroids Metakatz metaSpheres MetaSpheres Mickey Degods Millionaire Apes Club Mindfolk Mini Royale: Nations - Season 1 (Premium) Mob Monkettez MogulWars Monkey Ball MonkeyBall Gen Zero My name Is Sol My Name is Sol Myopa Mystic Potion Neopets Metaverse NFT Poetry NFTrees Solana NGMIPandas Nifty Nanas NON-FUNGIBLE “BEES” Nyan Heroes OGbottles Oink Club NFT OinkClub NFT PEEPS Pengu Love Personify Pesky Ice Cube Phantom Pilgrim Society PIMP MY THUG Pirates of Sol Bay - Bottles Pirates of Sol Bay - Treasures pitcrew Pix World NFT Pixel Island NFT PixelWorms PixWorldNFT Platypusol Family Playground Waves Playground: Waves PopsicleNFT Posh Dolphs Powder Heroes Prickly Pete’s Platoon - OG Cactoon Series PSY Network | PlanetZ Pudgy Pigeons Rabbit Punks Realm Kings Red paperclip RowdyRex Rug Toadz Ruled by Randomness: The Genesis SantaClaus Savage Dray by Squeak Brigade SavagesTotsys SawBunny Secret Duck Society SGF United Shadowy Super Coder Shadowy Super Coder DAO SharkBros shatteredmarble ShroomZ Slimeballz sLoot Smileys Smolpenguins Snek Gang soAlien SocksOnSolana Sol Diamond Hands Sol Lions SOL NFL PLAYER’S SOL NFL Players SOL Parasites Sol Slugs Sol Tamagotchi Sol Tapes SOLadies Solagon SolAlbums Solamids Solana Baby Monkey Business Solana Bananas Solana Birbs Solana Bros Solana Cat Gang Solana Fan The Game of Squid Solana Feline Business Solana Havana Cigar Club Solana Locks Solana Mystery Box Items Solana Mystery Items Solana Pickles Solana Reversed Monkey Business Solana Robot Business Solana Samurai Journey Solana Slugs SOLANA SUPERCAR CLUB Solana Surfers Solana Tactical RPG STACC solanabets | The Clique NFT SolArc NFT Solarnauts: Mission Bravo SolBlocks Solbusters Solccoons SolCrocos Soldalas SolDice SOLDIER RABBITS SolEmoji Solez SolFoxes SolGalaxy SolGangsta Solloons Sollyfish Solmon Solmoverse: Collection 0 Solmushies SOLNANA SolNauts SolOrbs Solryx SolSlimes SolSneakers Solsteads Surreal Estate SolStoners SolTowers Solutions Solvaders SolWatchers Soul Dogs Soulofox Space Bums Space Bums: Galaxy Mint Pass Spiderverse Spirits of Solana Squareheadz Squid Society Squirrelz Stash StratosNFT Structs Superballz Surging Bulls Synesthesia, by Labyrinth Terrarium Tanks Test Guys Test Guys Item Outpost The Assembly The Baby Boogles The Beverly Hills Car Club The Collectoooooor The Elementies The Exiled Apes The Nasty Boys The Rock theBULL by metaCOLLECTIVE TheDragonClub Thirsty Cactus Garden Party Thoughtful Folk NFT ThugDragonz Tiny Tigers Titanz ToneBox Undead Sols Vale Unleashed Rel Vampires of SOL Vampires Of SOL WallStreetPunkS Wicked Pigeon Posse Wieners Wieners Club Wildfire Native Winter Tiny Tigers Wolves On Wallstreet WOOFers World of Deities NFT WUKONGSOL Xperiment

Bug Bounty

In conjunction with this vulnerability research, Metaplex has launched a bug bounty program.

Earlier this week the @Metaplex Foundation announced a Bug Bounty program—our commitment to white-hat developers we’ve been spinning up for months.

Our first contributor, @jonluca, uncovered a vulnerability in CMv1 back in January. More below. 👇 https://t.co/sq0cjtLtTj

— Metaplex (@metaplex) March 18, 2022

Timeline

• Dec 31st - Fix for CMv2 is landed

• Tue Jan 04 2022 05:57:11 GMT-0500 - First attacker transaction is executed

• Tue Jan 04 2022 06:20:29 GMT-0500 - First transaction that tries to interact with the newly updated contracted is executed

• Tue Jan 04 2022 06:49:27 GMT-0500 - Last transaction that tries to interact with the newly updated contracted is executed

• Tue Jan 06, 2022, 18:25 GMT-0500 - Fix for CMv1 is landed

• Tue Jan 15, 2022, 21:15 GMT-0500 - Metaplex is alerted to this specific vulnerability.

• Fri Mar 11, 2022 - Metaplex bug bounty program is launched in conjunction with this post

• Fri Mar 18, 2022 - Metaplex bug bounty for CMv1 is announced

Appendix

This vulnerability was discussed in Discord’s and on Twitter but was not widely analyzed.

All the code for this research will be made public pending final vulnerability disclosures on various exchanges.

Screenshot from Seated app of a block in the LES in NYC

The rebates actually get pretty substantial - up to 50% in some cases. Half off dinner in NYC is a pretty enticing proposition (although most of the restaurants seem to be in the 10 - 20% range).

It’s a pretty interesting app - I’m not entirely sure what their business model is (as I can’t image that a known low-margin industry like restaurants would be able to rebate 50% of the total value of a bill), but it does work and their rebates are easy to claim.

I wanted to see if I could grab all their restaurants and visualize the stats on them.

Finding their API

I used the same methodology as in the previous article - luckily, the app didn’t do any form of TLS pinning, and I was able to quickly reverse their routes.

I then converted it to Python using a handy curl converter.

import requests

headers = {
    'Host': 'api.seatedapp.io',
    'Content-Type': 'application/json',
    'Flavor': 'native_app_v1',
    'Accept': '*/*',
    'Authorization': f"Bearer {os.environ.get('BEARER_TOKEN')}",
    'Accept-Language': 'en-US,en;q=0.9',
    'Accept-Encoding': 'gzip, deflate',
    'Platform': 'ios',
    'User-Agent': 'Seated/1075 CFNetwork/1327.0.4 Darwin/21.2.0',
    'Device': '?unrecognized?',
    'Build': '1075',
}

params = (
    ('city', '2'),
    ('isRemoveFutureWalkInVariant', 'false'),
    ('latitude', '40.71'),
    ('longitude', '-73.98'),
    ('mapLatitude', '40.71'),
    ('mapLongitude', '-73.99'),
    ('maxSeats', '2'),
    ('minSeats', '2'),
    ('page', '1'),
    ('size', '50'),
    ('slotForDate', '2022-01-02T18:30:00.000Z'),
)

response = requests.get('https://api.seatedapp.io/v2/search/map/dinein', headers=headers, params=params, verify=False)

I then adapted this to save all the restaurants for later processing, and played around with the size of the response (it seemed to error out on values above 400).

import requests
import json
import os

headers = {
  'Host': 'api.seatedapp.io',
  'Content-Type': 'application/json',
  'Flavor': 'native_app_v1',
  'Accept': '*/*',
  'Authorization': f"Bearer {os.environ.get('BEARER_TOKEN')}",
  'Accept-Language': 'en-US,en;q=0.9',
  'Accept-Encoding': 'gzip, deflate',
  'Platform': 'ios',
  'User-Agent': 'Seated/1075 CFNetwork/1327.0.4 Darwin/21.2.0',
  'Device': '?unrecognized?',
  'Build': '1075',
}


def get_params_from_page(page=1):
  return (
    ('city', '2'),
    ('isRemoveFutureWalkInVariant', 'false'),
    ('latitude', '40.71'),
    ('longitude', '-73.98'),
    ('mapLatitude', '40.71'),
    ('mapLongitude', '-73.99'),
    ('maxSeats', '2'),
    ('minSeats', '2'),
    ('page', str(page)),
    ('size', '400'),
    ('slotForDate', '2022-01-02T18:30:00.000Z'),
  )


restaurants = []
page = 1
while True:
  params = get_params_from_page(page)
  response = requests.get('https://api.seatedapp.io/v2/search/map/dinein', headers=headers, params=params)
  data = response.json()
  if 'restaurants' in data:
    restaurants += data['restaurants']
  else:
    print("Something went wrong")
  if 'metaData' in data and not data['metaData']['hasMoreItems']:
    break
  print(f"Completed page {str(page)}")
  page += 1

print(f"Found {len(restaurants)} restaurants")
with open('restaurants.json', 'w') as out:
  out.write(json.dumps(restaurants))
  out.close()

Each restaurant entry had the following shape:

{
  "id": 6673,
  "name": "Memphis Seoul",
  "longitude": -73.9579086303711,
  "latitude": 40.67172622680664,
  "priceRating": 2,
  "neighborhood": {
    "id": 92,
    "name": "Crown Heights ",
    "branchNavigationLink": "https://seated.app.link/8kpY6fF558",
    "cityId": 2
  },
  "image": {
    "url": "https://storage.googleapis.com/voco_main_bucket/images/3382905/original/19b89f300df44f21.png"
  },
  "images": [
    {
      "url": "https://storage.googleapis.com/voco_main_bucket/images/3382911/original/df2174d0b63060e8.png"
    }
  ],
  "isSurging": true,
  "yelpRating": 3.5,
  "baseReward": 32,
  "isWalkInOnly": true,
  "yelpBusinessUrl": "https://www.yelp.com/biz/memphis-seoul-brooklyn",
  "menuUrl": "https://www.getmemphisseoul.com/menus/",
  "address": "569 Lincoln Pl, Brooklyn, NY 11238, USA",
  "thoughts": "\"Our thoroughly unique menu is a tasty selection of Southern favorites fused with Korean ingredients and influences. Come for the savory pulled pork and Korean BBQ Meatloaf, then stay for the scrumptious ",
  "deliveryUrl": "https://direct.chownow.com/order/14302/locations/20081",
  "pickupUrl": "https://direct.chownow.com/order/14302/locations/20081",
  "deliveryReward": 25,
  "pickupReward": 25,
  "maxReward": 47,
  "phoneNumbers": [
    {
      "phoneNumber": "3473492561",
      "phonePrefix": "1"
    }
  ],
  "providerName": "ChowNow",
  "orderProviderType": "ONLINE",
  "isDeliverySurging": true,
  "isPickupSurging": true,
  "branchNavigationLink": "https://seated.app.link/tRSa5nQEy4",
  "primaryCuisine": {
    "id": 1,
    "name": "American",
    "branchNavigationLink": "https://seated.app.link/H0lPkpGUf4"
  },
  "maxPartySize": 12,
  "isOpen": true,
  "isDineIn": false,
  "availableSeatingOption": "UNKNOWN",
  "city": {
    "id": 2,
    "name": "New York City",
    "nameLower": "new york city"
  },
  "delivery": true,
  "pickup": true
}

Rewards

Once I had all the data, I wanted to see where most restaurants fell, in terms of their max reward distribution.

Rebate % # Restaurants (0, 5] 1 (5, 10] 289 (10, 15] 547 (15, 20] 266 (20, 25] 264 (25, 30] 116 (30, 35] 47 (35, 40] 3 (40, 45] 3 (45, 50] 2

Most restaurants fall in the 10-15% back, and only a handful are above 35%. I expected a more normal distribution, and fewer restaurants offering 20%+, but it’s a pretty healthy range and they’re clustered in a relatively high return zone.

Visualizing locations

I also wanted to see where all their restaurants were - I didn’t necessarily want to travel to Yonkers for 30% off a pizza.

This worked very well - it was trivial to then cut the dataframe to only restaurants in downtown NYC:

les_df = geo_df[geo_df['latitude'].between(40.7,40.767645)]
les_df[['name',reward]].head(10)

Rating vs Reward

What’s interesting is that every restaurant also has a yelp rating. I wondered if the rating was correlated to the rebate size - were worse restaurants offering more (if it was even the restaurant’s choice on the max reward %?) due to low rewards?

And the answer is… kinda? There highest rewards show up at the 3 and 4 star ratings, but they also have the highest distribution of restaurants, so with more data points you’re more likely to have outliers.

The above shows the stats for the reward based on the yelp rating. If we assume that 1, 2 and 5 star ratings are low-volume outliers, we see a slight correlation, but not enough to draw any conclusions from.

Reward vs Price

Another interesting potential correlation is with the maximum rebate % and how expensive the restaurant is - you’d imagine that nicer restaurants would offer less back.

This is somewhat observed, although there are some 4 dollar sign restaurants that offer 30% back.

Code

The code for the querying and analysis can be found on GitHub, at this repo.

Joining Seated

If you end up signing up for Seated, you can use my referral code jonluca1 to get $15 bonus on your first meal.

I wanted to see if it would be possible to set up a script to monitor for changes, and to alert me any time there was any availability. There was one restaurant in particular, Chef’s Table at Brooklyn Fare, that I wanted to book for early next year.

Finding their API routes

Getting availability

I fired up BurpSuite and set it up. I navigated to their restaurant page and was happy to see that there wasn’t any certificate stapling going on, and the response was clean JSON (the recent rise of protobufs and graphql have made clean API discovery a bit harder).

Fortunately for us, it seemed like there was a pretty clean API to fetch the availability of any restaurant - a simple PUT to https://mobile-api.opentable.com/api/v3/restaurant/availability request with the following body:

{
  "forceNextAvailable": "true",
  "includeNextAvailable": true,
  "availabilityToken": "<jwt>",
  "dateTime": "2021-12-03T19:00",
  "requestTicket": "true",
  "allowPop": true,
  "attribution": {
    "partnerId": "84"
  },
  "partySize": 2,
  "includeOffers": true,
  "requestPremium": "true",
  "requestDateMessages": true,
  "rids": ["211123"],
  "requestAttributeTables": "true"
}

And the response had exactly what we were looking for.

{
  "dateTime": "2021-12-03T19:00",
  "availability": {
    "dateTime": "2021-12-03T19:00",
    "noTimesReasons": ["NoTimesExist"],
    "minPartySize": 2,
    "maxPartySize": 6,
    "maxDaysInAdvance": 42,
    "id": "211123",
    "timeslots": [],
    "dateMessages": [],
    "token": "eyJ2IjoyLCJtIjowLCJwIjowLCJzIjowLCJuIjowfQ",
    "hasPremiumTables": false,
    "availabilityToken": "eyJ2IjoyLCJtIjowLCJwIjowLCJzIjowLCJuIjowfQ"
  },
  "suggestedAvailability": ["... entry availability"],
  "experienceList": {
    "results": []
  },
  "hasPremiumTables": false,
  "dayAvailability": {}
}

A quick conversion to python then gets us:

import requests
import json

cookies = {
'Some Cookies Here': '...'
}

headers = {
    'Host': 'mobile-api.opentable.com',
    'Accept': 'application/json',
    'Content-Type': 'application/json',
    'X-Ot-Sessionid': 'SessId',
    'Accept-Encoding': 'gzip, deflate',
    'Authorization': 'Bearer <TOKEN>',
    'User-Agent': 'com.contextoptional.OpenTable/15.2.0.16; iPhone; iOS/15.1.1; 3.0;',
    'Content-Length': '415',
    'Accept-Language': 'en-US;q=1.0,en-GB;q=0.7,en;q=0.5,*;q=0.3',
    'Connection': 'close',
}

data = {
    "forceNextAvailable": "true",
    "includeNextAvailable": True,
    "availabilityToken": "<jwt>",
    "dateTime": "2021-12-03T19:00",
    "requestTicket": "true",
    "allowPop": True,
    "attribution": {
        "partnerId": "84"
    },
    "partySize": 2,
    "includeOffers": True,
    "requestPremium": "true",
    "requestDateMessages": True,
    "rids": [
        "211123"
    ],
    "requestAttributeTables": "true"
}

response = requests.put('https://mobile-api.opentable.com/api/v3/restaurant/availability', headers=headers, cookies=cookies, data=json.dumps(data), verify=False)

Success! I then tried removing all non essential cookies and headers, and it still worked - the only thing that mattered were the Authorization and Content-Type request headers.

Another interesting thing was the availabilityToken in the request body. It clearly looked like a jwt (any time you see ey in a string your first thought should be jwt, much like a string ending in = should make you think base64), but when I put in a JWT debugger it looked like a simple flag mechanism.

{
  "v": 2,
  "m": 1,
  "p": 1,
  "s": 0,
  "n": 0
}

My request to Chef’s Table at Brooklyn Fare had m and p set to 0, whereas my request for another restaurant with a lot of availability had them set to 1. Swapping out the jwt’s led to the same response, just with a different token for the available bookings.

The time slots were an array of objects with this format:

{
  "dateTime": "2021-12-06T15:30",
  "available": true,
  "redemptionTier": "GreatDeal",
  "diningAreas": [
    {
      "id": "1",
      "isDefaultArea": true,
      "availableAttributes": ["default", "outdoor"]
    }
  ],
  "token": "<jwt>",
  "slotHash": "<hash>",
  "points": 100,
  "type": "Standard",
  "attributes": ["default", "outdoor"],
  "priceAmount": 0
}

The slotHash is the unique ID for this given reservation.

Booking a time

I made a dummy booking with a restaurant that had loads of availability. When you go through their flow, they first fire off a reservation “lock”, that guantees you that spot, and once you’ve finished whatever questionnaire the restaurant might have (seating preferences, etc), it fires off another request to complete the reservation.

The lock fires a POST to /api/v1/reservation/<restaurant_id>/lock, with this body:

{
  "partySize": 2,
  "dateTime": "2021-12-06T15:30",
  "selectedDiningArea": {
    "tableAttribute": "default",
    "diningAreaId": "1"
  },
  "hash": "<hash>",
  "attribution": {
    "partnerId": "84"
  }
}

If your request is valid and that slot has not been locked by someone else, the response will now include a unique ID that corresponds to that reservation:

{
  "id": "<numericIdOfReservation>",
  "rid": "19306",
  "date": "2021-12-06T15:30",
  "partySize": 2,
  "offerLockId": "0",
  "stripeKey": "pk_live_<STRIPEKEY>",
  "creditCardCancellationPolicy": {
    "cancellable": true,
    "cancellationCutoffDate": "2021-12-04T17:00"
  },
  "occasions": [
    "birthday",
    "anniversary",
    "date",
    "special_occasion",
    "business_meal"
  ]
}

The final step is to fire one last POST request to /api/v1/reservation/<restaurant_id> with a much larger body:

{
  "diningFormOptIn": true,
  "partySize": 2,
  "gpid": "<GPID>",
  "countryId": "US",
  "attribution": {
    "partnerId": "84"
  },
  "loyaltyProgramOptIn": true,
  "optIns": {
    "smsNotifications": {
      "reservationSms": true,
      "waitlistSms": false
    },
    "openTableDataSharing": {
      "businessPartners": true,
      "corporateGroup": true,
      "pointOfSale": true
    },
    "dataSharing": {
      "guestShare": true,
      "dinerProfileShare": true,
      "sync": true
    },
    "restaurantEmailMarketing": {
      "restaurantEmails": true
    },
    "emailNotifications": {
      "diningFeedback": true
    },
    "emailMarketing": {
      "newHot": false,
      "restaurantWeek": false,
      "spotlight": false,
      "product": false,
      "promotional": false,
      "insider": false,
      "dinersChoice": false
    }
  },
  "rewardTier": "GreatDeal",
  "campaignId": "7",
  "hash": "<HASH>",
  "points": 100,
  "loadInvitations": false,
  "number": "<PHONENUMBER>",
  "notes": "",
  "slotAvailabilityToken": "<TOKEN>",
  "selectedDiningArea": {
    "diningAreaId": "1",
    "tableAttribute": "default"
  },
  "lockId": "PreviouslyObtainedLockId",
  "dinerId": "<DINERID>",
  "location": {
    "latitude": 40.74,
    "longitude": -73.98
  },
  "dateTime": "2021-12-06T15:30",
  "uberUuid": "<UUID>"
}

Canceling a reservation

Not that I’d necessarily want to cancel a reservation, but it looks as easy as a DELETE request to DELETE /api/v3/reservation/<restaurant_id>/<confirmation_number>

Hooking it all up

I put it all together into a python script, which be found on GitHub. This will take in a restaurant ID and your time range, and automatically book it once it becomes available.

Finally, just for the hell of it, I hooked it up to twilio so that it would send a text message once the reservation had been booked. It will now send me a text message once a table has been booked, and save the details to a file so that it won’t double book a reservation.

I tested it and it worked great. I’ve got it monitoring a few different restaurants right now, and it’ll book it as soon as someone cancels or it becomes available, or the restaurant releases availability. I still need to modify it so that it works with restaurants that require a credit card transaction, but that can be saved for the next post.

The code can be found on GitHub. There isn’t a way to fetch your bearer token automatically right now, or search for a restaurant, so you’ll need to know those ahead of time to make this work, but you can follow the steps above (or open a PR!).

Future work

The next site I want to automate is Tock, as they have most of the best restaurants in NY, but their endpoints are protobufs, which are a bit harder to reverse engineer. Resy is also an option, but with the Amex acquisition it’s actually pretty easy to get reservations at sold out restaurants if you have an Amex Platinum card.

The former is what most traditional scraping looks like - you manually inspect a web page, you determine the right xpath/css selectors to follow, and then instruct a scraper to statically request the page and scrape it.

The rise of SPAs has made this approach a bit less impractical - you now have to actually render the content, which is is significantly slower, to have the content you want to scrape show up in the DOM.

At the same time, it made web scraping as a whole much easier - sites that used to be server rendered now have nice, machine readable routes that serve JSON. This has lead to tools like Burp Suite and Charles Proxy being coopted from their original use of finding security vulnerabilities to being primarily used for web scraping.

In this article I want to introduce a third, more niche attack surface for scraping web pages - scraping through memory snapshots and their respective dominator graph. It’s particularly applicable when the client is encrypting its network requests and responses.

Chrome Devtools

Devtools has a nice feature to detect memory leaks - the Memory tab.

It will take a snapshot of the current tabs memory, and built a graph out of it to inspect.

The memory tab is mostly used to compare different memory snapshots - you take a snapshot, perform some actions on the page, take another snapshot, and look at everything that got allocated to see if there are any memory leaks.

Since it’s pretty much a full memory snapshot, it stores all strings and objects - which, for something like React or Vue, will include all props and (typically) network request responses.

Memory snapshots

Inspecting memory isn’t new in the reverse engineering space - for native apps, it’s one of the most common ways of finding out what an application is doing and how its internal logic is structured.

Within the web world, though, it’s hardly ever used. Most of the literature online is regarding finding memory leaks. All the tools and software around it are therefore built around that - they don’t care about the contents of the memory, they care about the origination source/allocator and the references/dominators to it.

When does this make sense?

Using memory snapshots makes sense in two cases - 1), when the client encrypts the network response, and you don’t have the time or energy to reverse the bundle to understand how to decrypt it programatically (like in Blind’s case, which I did here) and 2) when the client requests are machine readable, but there are multiple requests that get stitched together in memory to create the desired objects.

Reversing the spec

Saving a memory snapshot generates a heapsnapshot file - this is a JSON file with a given format, that is not document anywhere as far as I can tell. These snapshots are the same in Node and in Chrome, and all documentation for both just tell you to upload it to the memory tab in a new devtools instance to view it.

Luckily the code for devtools is public, so I managed to reverse it and turn it into a CLI tool - this accepts heapsnapshots as inputs, and can parse them into the graph.

Extracting JSON

At this point, you can just go through and try and parse every string in memory as JSON, and see what sticks around. This will be an easy way to find in memory JSON strings - however, these aren’t as common as you might think. The v8 compiler is pretty efficient in knowing what to keep in memory and what to discard, so it’ll typically just keep the parsed objects.

Reconstructing objects

You can reconstruct objects in memory by traversing through the graph and stitching the nodes together - everything ends up being a primitive, and graph nodes can just contain properties, so they’re proxies for objects.

It’s useful to sort by retained size (which the code in the repo has a function for), and look at the heaviest objects - these are typically going to be your most interesting allocations, especailly if you perform an allocation recording.

Future plans

The code for the project can be found here, which will parse any chrome or node heapsnapshot.. In part 2 I’m going to add the ability to fully reconstruct objects from this graph.

Function.length

Call Function.length will return the number of arguments a function is expecting.

The spread operator will not be included in the count.

Array.map(func)

when you call Array.map(func), the mapped function gets called with 3 arguments, not just the value.

So for:

["10", "11", "12"].map(parseInt);

You’d expect to get

[10, 11, 12];

but in reality get

[10, NaN, 1];

This is because .map(parseInt) calls the function with three arguments - (currentValue, index, array). This is normally not an issue, but becomes an issue when the mapped function takes additional arguments that do not correspond to the ones being passed in.

parseInt takes in two arguments - value, [, radix], and thus tries to parse 11 with radix 1, which is NaN

Values are truth-y by default

The only falsey values are:

[0, -0, 0n, "", "", null, undefined, NaN, false];

Everything else is truthy - including [], an empty Set(), and an empty object.

Null comparisons to 0

I ran into a nasty bug once where a value I thought was guaranteed to be a number was actually explicitly set to null. I was doing a comparison with 0 and ran into this weird behavior:

Array.sort sorts by string sequence code

Call .sort() on an array of numbers will not sort them numerically. Which is perplexing

Null is not equal to zero, and is not greater than zero, but is greater than or equal to zero.

String.replace() only replaces the first instance of a match

It’s almost a rite of passage for javascript developers to be bewildered that String.replace only replaces the first instance of a match in a string.

Thankfully in ES2021 we now have String.replaceAll, which behaves as you’d expect.

WTFJS

Someone also pointed out WTFJS.com which is a site that has a lot more javascript oddities and examples.

I’ve gone through a few generations - I’ll use this post to catalogue where I started and what I’m doing now. If you want to skip the post and just see the final code, it can be found here.

Gen 1

Generation one was trusty old requests. Need to make 10 requests? Wrap it in a for loop and make them iteratively.

import requests

results = {}
for i in range(10):
    resp = requests.get('https://jsonplaceholder.typicode.com/todos/1')
    results[i] = resp.json()

This isn’t bad - 40 requests in 2.8s, or 1 req/70ms. Not an issue at all. This is fine when you need to bruteforce a 3 digit passcode - you can get 1000 requests done in 70s. Not great, but fast enough, and no need for external libraries or any research.

As soon as you get to something in the 4 character range, though, this becomes unwieldy.

Gen 2

The next step here was to find ways to make these requests using Threads. Spin off a native thread for each request, and let them run behind the scenes.

Set up a queue and pool to pull URLs from, and we’re good. The queue and Worker threads are defined pretty simply below:

from queue import Queue

from threading import Thread


class Worker(Thread):
  """ Thread executing tasks from a given tasks queue """

  def __init__(self, tasks):
    Thread.__init__(self)
    self.tasks = tasks
    self.daemon = True
    self.start()

  def run(self):
    while True:
      func, args, kargs = self.tasks.get()
      try:
        func(*args, **kargs)
      except Exception as e:
        # An exception happened in this thread
        print(e)
      finally:
        # Mark this task as done, whether an exception happened or not
        self.tasks.task_done()


class ThreadPool:
  """ Pool of threads consuming tasks from a queue """

  def __init__(self, num_threads):
    self.tasks = Queue(num_threads)
    for _ in range(num_threads):
      Worker(self.tasks)

  def add_task(self, func, *args, **kargs):
    """ Add a task to the queue """
    self.tasks.put((func, args, kargs))

  def map(self, func, args_list):
    """ Add a list of tasks to the queue """
    for args in args_list:
      self.add_task(func, args)

  def wait_completion(self):
    """ Wait for completion of all the tasks in the queue """
    self.tasks.join()

And the actual query code is fairly straightforward - just define a function that’ll populate a global variable using some unique ID, and have it make the request off in its own thread.

urls = [f"https://jsonplaceholder.typicode.com/todos/{i}" for i in range(40)]
pool = ThreadPool(40)

r = requests.session()


def get(url):
    resp = r.get(url)
    results[i] = resp.json()


pool.map(get, urls)
pool.wait_completion()

Now we’re getting somewhere - 40 requests in 365ms, or 9.125ms per request. The same 1000 requests that would’ve taken 1m10s earlier now finishes in a little over nine seconds, or just about a 7x speed up. Not bad for a pretty naive implementation of threading. Can we get it even faster, though?

Gen 3

A few years back I was introduced to the library aiohttp - which is Asynchronous HTTP Client/Server for asyncio and Python. This leverages the new (at the time) async capabilities of python and lose the actual overhead of Threads.

There is a bit of monkey patching I’ve had to do to make it work with all the various request types - specifically around its conformance to the cookie spec and to get it to work properly in jupyter notebook, where I like to play around with a lot of network requests.

Once we start looking at a pool of thousands of requests, we also want to be able to throttle ourselves - our laptops can only open so many TCP connections at once, and fire off so many bits in a given second. I defined a helper called gather_with_concurrency - it’s a way of using asyncios gather with a semaphore, to limit the amount of tasks we work on in any given second.

This slows us down a bit due to the semaphore overhead, but if you’re doing anything in the alphanumeric 4 digit space you’re looking at 36^4, or 1.7m requests, which should probably be pooled and throttled a bit, and the overhead from the semaphore is worth it.

async def gather_with_concurrency(n, *tasks):
    semaphore = asyncio.Semaphore(n)

    async def sem_task(task):
        async with semaphore:
            return await task

    return await asyncio.gather(*(sem_task(task) for task in tasks))

Next we set up the connector and the custom session

conn = aiohttp.TCPConnector(limit=None, ttl_dns_cache=300)
session = aiohttp.ClientSession(connector=conn)

Here we limit each host to none - it won’t throttle itself internally, since we want to control that externally. We also bump up the dns cache TTL. For the purposes of this blog post this won’t matter, but by default it’s 10s, which saves us from the occasional DNS query.

Finally we define our actual async function, which should look pretty familiar if you’re already used to requests. We also disable SSL verification for that slight speed boost as well.

async def get(url):
    async with session.get(url, ssl=False) as response:
        obj = await response.read()
        all_offers[url] = obj

Now we’re really going! 40 requests in 100ms, or 4ms per requests. We can do about 250 requests per second - however, at this speed, the overhead of the initial function set up and jupyter notebook is actually a significant portion of the overall cost.

If we bump it up to 4000 requests, we see that we actually get closer to a 1.574s execution time, or about 56% of the time it took us to make 10 requests iteratively.

We can make one request every 0.393ms, or 393 microseconds. We can blast through the entire alphanumeric space for a 4 character permutation in about 660 million microseconds, or 11 minutes, all from my MacBook pro.

Our Threading implementation also benefits from the increase in pool - giving it a 100 threads (the same as the semaphore in the asyncio version) gives us a time to completion of 8s for 4000 urls, or a little over 2ms per URL. Nowhere near our aiohttp implementation but not terrible either.

The same 36^4 requests using the ThreadPool would take 48 minutes, though.

We can clean up the code and optimize it slightly as well:

import sys
import os
import json
import asyncio
import aiohttp


# Initialize connection pool
conn = aiohttp.TCPConnector(limit_per_host=100, limit=0, ttl_dns_cache=300)
PARALLEL_REQUESTS = 100
results = []
urls = ['https://jsonplaceholder.typicode.com/todos/1' for i in range(4000)] #array of urls

async def gather_with_concurrency(n):
    semaphore = asyncio.Semaphore(n)
    session = aiohttp.ClientSession(connector=conn)

    # heres the logic for the generator
    async def get(url):
        async with semaphore:
            async with session.get(url, ssl=False) as response:
                obj = json.loads(await response.read())
                results.append(obj)
    await asyncio.gather(*(get(url) for url in urls))
    await session.close()

loop = asyncio.get_event_loop()
loop.run_until_complete(gather_with_concurrency(PARALLEL_REQUESTS))
conn.close()

print(f"Completed {len(urls)} requests with {len(results)} results")

Optimal semaphore size?

If we bump our concurrent requests to 4k we see a drastic loss in performance.

This is nearly a 3x slow down due to resource contention issues locally.

The optimal number will depend on your host - beefier set ups will have higher concurrency limits, and if you’re running this on a remote host on something like digital ocean you can crank this number up quite a bit.

Interestingly enough the optimal semaphore value was right around 60.

I mostly do this locally at home, though, for my side projects - introducing parallelization and multiple hosts can get you numbers that are an order of magnituded better than this, but the purpose of this excercise is seeing what we can hit on a local machine with jupyter notebook.

Gen 4

Know of a faster implementation of an HTTP library that is stateful and works locally? Let me know!

HTTPX

Update - 6/17/21

A poster on lobste.rs said that I should try out httpx. HTTPX is a modern implementation of a python web client.

Unfortunately, in my testing, it was strictly slower than aiohttp. I used their async library with the same sempahore restricting the number of processes ran, but it was still slower.

I also tried a native gather, with punting the concurrency down to the library - this did not help either.

PyCurl

Someone on that same lobste.er’s thread suggested pycurl.

PyCurl is different in that it feels like a pretty raw wrapper to curl. Writing the code felt more like dispatching actions and opening sockets than dealing with a nice http library.

The results were impressive, but the aiohttp library was still faster. This was my first time writing a pycurl implementation, though, based on this template - introducing native threading might be able to speed it up, but I still haven’t seen anything faster than the 393 microseconds approach.

If you know how to set up HTTPX or PyCurl in a way that’s faster let me know!

UVLoop

Addendum: 8/27/21 - I received an email from Steve telling me about uvloop, a faster, drop in replacement for asyncio’s event loop.

It doesn’t seem to have impacted the performance all that much - it did have lower variance, though. Across multiple runs of a regular asyncio event loop, I would get as high as 3s for the same 4000 requests; with uvloop, it never broke 2.1s.

For a drop in replacement it seems pretty great - I don’t think it’ll help much at this stage, though, because most of the timing is due to the actual network call at this point. The threading and event loop implementation isn’t adding that much overhead, I’m guessing.

Prerequisites

Roughly 250,000 GitHub users qualified. You must have had at least 15 followers during the week of 2019-02-04. If you didn’t have a GitHub account at the time, or didn’t have 15 followers, or didn’t have a valid public SSH key uploaded, you were not eligible for the airdrop.

First steps

Download hs-airdrop and register for an account on https://namebase.io. Namebase will create a free HNS wallet for you, which you can easily transfer to BTC.

git clone https://github.com/handshake-org/hs-airdrop.git && cd hs-airdrop && npm install

Next go to https://www.namebase.io/ and find your wallets address.

To redeem your HNS you’ll need:

• The hs-airdrop binary

• The path to your private key

• Your wallet address

Namebase also has similar instructions on redeeming these coins at https://www.namebase.io/airdrop.

SSH

If you want to use SSH, first confirm that the key you want to use is registered on GitHub, and was associated with your account on February 4th, 2019.

List all your keys with

$ ls ~/.ssh/

Most likely it will be named id_rsa. Then run the hs-airdrop binary with the path to your private key.

./bin/hs-airdrop --bare ~/.ssh/id_rsa hs1...<your address from namebase> 0.1

It will ask you to decrypt your private key. Note that the private key is never sent anywhere - you can verify this yourself from the source code of hs-airdrop, here: https://github.com/handshake-org/hs-airdrop. This will then proceed to submit your confirmation and check if it is in the airdrop tree. If it is, you’ll get a base64 confirmation, which you’ll submit below.

PGP

PGP is a little more tricky - the documentation on the repo is a bit lacking.

On MacOS, using PGP Suite, you can do the following.

First, get all your available keys:

$ gpg --list-keys
/Users/jonlucadecaro/.gnupg/pubring.kbx
---------------------------------------
pub   rsa4096 2017-08-24 [SC] [expires: 2033-08-20]
      849E61D17094A964866AE510E6DC4811DD593AC7
uid           [ultimate] JonLuca De Caro <jonlucadecaro96@gmail.com>
sub   rsa4096 2017-08-24 [E] [expires: 2033-08-20]

The key ID is the value below the pub key (in the case of the above, 849E61D17094A964866AE510E6DC4811DD593AC7). You now want to export the private key, replacing my public key ID with yours.

$ gpg --armor --export-secret-keys 849E61D17094A964866AE510E6DC4811DD593AC7 > ~/Desktop/sec.asc

It will ask you for the keys password.

to check if your key is in the airdrop tree, you next run, again replacing my key with yours:

./bin/hs-airdrop --bare ~/Desktop/sec.asc 849E61D17094A964866AE510E6DC4811DD593AC7 hs1...<your address> 0.1

This will tell you if your key is in the airdrop tree, and if it is, it will print out a base64 string with your airdrop redemption.

Submitting your confirmation

If your SSH or public key was in the airdrop tree, it will print a base64 string. Go to https://www.namebase.io/airdrop and paste it into the 5th box.

If it was valid, you’ll receive your HNS in roughly 16 hours.

You can then proceed to sell it on namebase.

Just paste in whatever BTC balance you want to send it to, and it’ll show up in a few minutes to hours. You can also use the coins to bid on names in namespace, or to support additional FOSS projects!

Multiple keys

Note that if you have multiple SSH keys, it will only allow you to redeem for one. It will create valid nonces for each key, but once you redeem one in the chain, it invalidates the rest. If you submit the base64 for it it will look like it was accepted, but it will be rejected by the miners

Other Airdrops

• Keybase did an XLM drop worth about $1000 USD as of May 2021 - check your https://keybase.io account

• Uniswap did a 400 ETH drop worth about $16,000 USD as of May 2021 - https://airdrops.io/uniswap/