Now, there are a number of online applications to do that, but they have at least one of the following problems, fundamentally the last one:

• They pull your Twitter posts from the API, which only allows you to read at most the latest 200 tweets, so removing the older ones becomes impossible.
• Some of them get around this by asking you to upload your entire Twitter archive… which contains a lot more than just your tweets (i.e. your private messages). (EDIT: I’m being told this is no longer the case, now it just contains your public timeline)
• I don’t trust them.

So naturally I rolled my own. The code is crude but it worked for me. It uses the Twitter archive zip file as well, but since it works locally you don’t need to trust a third party with your personal data. With this I managed to delete over 60.000 tweets in a day and a half, roughly – it can’t be done much faster because of the API rate limiting, but then again, what’s the rush?

Now, to use it, follow these steps:

1. Get your Twitter archive. You can do this by going to Settings -> Account -> Your Twitter archive. This will send you an email with a download link to a zip file.
2. Get my script and place it in the same directory as the zip file you just downloaded.
3. Go to https://apps.twitter.com/ and create a new application. This will get you the consumer key and the consumer secret, take note of those.
4. Authorize your app to access your own account (you do it in the same place right after creating your new app). Now you have the access token key and secret, take note of those too.
5. Edit the script and add all those tokens and secrets at the beginning of the file. Add the name of the zip file as well. Since you’re at it, review the source code – you shouldn’t be running code you downloaded from some random blog without reading it first!
6. Run the script. This will take some time. Lock your PC. Get off the chair. Go out. Enjoy the real world. It’s the ultimate 3D experience, and it comes in HD!

Download

twitter_cleanup.py

Source

    #!/usr/bin/env python

    import anyjson
    import twitter
    import zipfile

    # Insert your API secrets here.
    api = twitter.Api(
        consumer_key='rhYvKSSciMHA9c8Dnp6OjzhRn',
        consumer_secret='1vRFTts73Mkaq2dgAGf5XYamKWHN10EihOH1zbJwp2U2TKzET1',
        access_token_key='37139784-SsXYm6jU5xqifX1hKiNfqynTEzRAoHKFPJfb1Wgbw',
        access_token_secret='bh71h9S17lvDxdRV6TrgAuNUlGc3EkwSDDSu8vcfBFn3n',
    )

    # Insert the name of the zip file with your Twitter archive here.
    twitter_archive = 'twitter.zip'

    # If you had to interrupt the script, just put the last
    # status ID here and it will resume from that point on.
    last = 0

    def read_fake_json(zip, filename):
        data = zip.open(filename, 'rU').read()
        first_line, data = data.split("\n", 1)
        first_line = first_line.split("=", 1)[1]
        data = first_line + "\n" + data
        return anyjson.deserialize(data)

    def parse_tweets_zipfile(filename):
        print "Parsing file: %s" % filename
        tweet_ids = {}
        with zipfile.ZipFile(filename, 'r') as zip:
            tweet_index = read_fake_json(zip, 'data/js/tweet_index.js')
            for item in tweet_index:
                tweets_this_month = read_fake_json(zip, item['file_name'])
                assert len(tweets_this_month) == item['tweet_count']
                tweet_ids["%d/%02d" % (item['year'], item['month'])] = [x['id'] for x in tweets_this_month]
        return tweet_ids

    if __name__ == "__main__":
        begin = False
        tweet_ids = parse_tweets_zipfile(twitter_archive)
        for date in sorted(tweet_ids.keys(), reverse=True):
            year, month = date.split("/")
            if int(year) < 2016:
                print "Deleting tweets from: %s" % date
                for tid in tweet_ids[date]:
                    if begin or last == 0 or tid == last:
                        begin = True
                        error_counter = 0
                        while True:
                            try:
                                api.DestroyStatus(tid)
                                print "%d: DELETED" % tid
                                break
                            except twitter.error.TwitterError, e:
                                try:
                                    message = e.message[0]['message']
                                    retry = False
                                except:
                                    message = repr(e.message)
                                    retry = True
                                print "%d: ERROR   %s" % (tid, message)
                                error_counter += 1
                                if error_counter > 5:
                                    print "Too many errors, aborting!"
                                    exit(1)
                                if not retry:
                                    break

In my previous post I briefly mentioned a rather creative use for URL shorteners: to store the contents of arbitrary files (a.k.a. “parasitic storage”). There was previous work on this area, namely a tool called TinyDisk that could implement a rudimentary filesystem on top of the TinyURL service. The bad news is, the site that hosted the tool appears to be down and I couldn’t get to play with it.

Abusing tinyurl.com

After a deep, thorough investigation (read: 30 seconds of googling) I came to the conclusion that no source code sample was available to do this. Naturally, the solution was to roll my own! It’s a crude proof-of-concept but it works. My first idea was to break up the file to be uploaded into small chunks that could fit the maximum length allowed in an URL, then ask TinyURL to shorten each of them, and keep the shortened URLs to be able to download the file later.

I started to test the tolerance of the TinyURL API and found three interesting things. One, that I wouldn’t even need to disguise the data as valid URLs, because TinyURL makes no validation of any kind – all I had to do was make sure the data was encoded in a way that wouldn’t break the HTTP requests. I chose to use hex encoding, but I’m sure there are more efficient encodings that would do the job nicely. Two, I could send obscenely large URLs. Sending 256 Kb of POST data worked like a charm so I chose that for the chunk size. The next power of two (512 Kb) caused the TinyURL server to drop the connection without a reply but I haven’t determined if this was really a limitation in their server or in my Squid proxy – in any case I thought it was a good idea to leave it at that. The third thing I noticed was that after a while of sending repeated requests the TinyURL service would stop responding for a while, I guess this is a protection against spammers. Quite understandable given the fact that this API is anonymous and doesn’t require an API key like many other services. The solution is to wait a little while when this happens before making any more HTTP requests.

Now the problem is how to get the data back. A simple GET request won’t work behind proxies, because TinyURL blindly returns our data in a Location: header and predictably my Squid refused to parse it when I tried. The solution I found was the preview feature. Changing the http://tinyurl.com/somecode into http://preview.tinyurl.com/somecode returns a webpage that contains the target URL – then we can parse the HTML to get the data back. It’s actually easier this way, because urllib2 tries to follow redirections by default and that would have called for some hacks on our part. Downloading the preview page is painless, but it also introduces more overhead (a 256 Kb block produces a preview page of a approx. 1 Mb).

That’s pretty much it… when we upload a file we break it into 256K blocks and send it to the TinyURL servers, then we store each returned short URL in a text file. Since all shortened URLs begin the same way (http://tinyurl.com/) we only need to store the code after the slash. The we download each block by asking for the preview page for each URL and extracting the data from the HTML page.

An example: let’s upload an image file (Plaza_Congreso__2_by_QvasiModo.jpg, 217,730 bytes).

  $ ./tinyurlfs.py upload Plaza_Congreso__2_by_QvasiModo.jpg Plaza_Congreso__2_by_QvasiModo.txt
  Created: http://tinyurl.com/yhbj6q6
  $ cat Plaza_Congreso__2_by_QvasiModo.txt
  yhbj6q6
  $

We can see it fit in a single 256K block, so we got only one shortened URL in response. The output text file only contains the code of the URL (7 bytes plus the newline character). Now let’s fetch it back and check the file contents are intact:

  $ ./tinyurlfs.py download Plaza_Congreso__2_by_QvasiModo.txt Plaza_Congreso__2_by_QvasiModo\ \(1\).jpg
  Reading: http://preview.tinyurl.com/yhbj6q6
  $ cmp -l Plaza_Congreso__2_by_QvasiModo.jpg Plaza_Congreso__2_by_QvasiModo\ \(1\).jpg 
  $

Let’s try the same with a larger file (BigBillBroonzy-BabyPleaseDontGo1.mp3, 3,915,875 bytes).

  $ ./tinyurlfs.py upload BigBillBroonzy-BabyPleaseDontGo1.mp3 BigBillBroonzy-BabyPleaseDontGo1.txt
  Created: http://tinyurl.com/y9kqqgk
  Created: http://tinyurl.com/ybkmjyc
  Created: http://tinyurl.com/yfx3xt6
  Created: http://tinyurl.com/yd8d4wf
  Created: http://tinyurl.com/y9ozqog
  Created: http://tinyurl.com/yjqj72c
  Created: http://tinyurl.com/ycclscw
  Created: http://tinyurl.com/yd2webd
  Created: http://tinyurl.com/ycmnp4v
  Created: http://tinyurl.com/yctbjom
  Created: http://tinyurl.com/y9n4d22
  Created: http://tinyurl.com/ybsrkfg
  Created: http://tinyurl.com/y9qoto5
  Created: http://tinyurl.com/yf2cp7o
  Created: http://tinyurl.com/yd94r9n
  $ ./tinyurlfs.py download BigBillBroonzy-BabyPleaseDontGo1.txt BigBillBroonzy-BabyPleaseDontGo1\ \(1\).mp3
  Reading: http://preview.tinyurl.com/y9kqqgk
  Reading: http://preview.tinyurl.com/ybkmjyc
  Reading: http://preview.tinyurl.com/yfx3xt6
  Reading: http://preview.tinyurl.com/yd8d4wf
  Reading: http://preview.tinyurl.com/y9ozqog
  Reading: http://preview.tinyurl.com/yjqj72c
  Reading: http://preview.tinyurl.com/ycclscw
  Reading: http://preview.tinyurl.com/yd2webd
  Reading: http://preview.tinyurl.com/ycmnp4v
  Reading: http://preview.tinyurl.com/yctbjom
  Reading: http://preview.tinyurl.com/y9n4d22
  Reading: http://preview.tinyurl.com/ybsrkfg
  Reading: http://preview.tinyurl.com/y9qoto5
  Reading: http://preview.tinyurl.com/yf2cp7o
  Reading: http://preview.tinyurl.com/yd94r9n
  $ cmp -l BigBillBroonzy-BabyPleaseDontGo1.mp3 BigBillBroonzy-BabyPleaseDontGo1\ \(1\).mp3
  $

Abusing ito.mx

TinyURL seems to be the best choice for this. But what about the other URL shortening services? Could they be similarly abused?

Our next target will be the Mexican service ito.mx. They appear to be running the same software as cru.ms, but possibly an older version. When comparing each API you can see they’re quite similar – and in fact I’ve used parameters in ito.mx that were only documented in cru.ms and they worked. I also found some checks in cru.ms to be missing from ito.mx, much to our advantage.

The key elements here are: 1) ito.mx lets you choose an arbitrary code for your short URLs, so we can use that to store data, and 2) it lets you create links to other ito.mx links. Also note that this service is not nearly as permissive as TinyURL: the long URLs must be well formed and begin with “http://” or “https://”, invalid characters must be escaped properly (unescaped invalid characters are automatically replaced by hyphens, we don’t want that), we need to make a short pause between HTTP requests (flood protection) and also for some odd reason it won’t let you shorten links to it’s competitor services (!). Try shortening http://bit.ly/test if you don’t believe me!

My idea to implement a file storage here was the following: use only the codes to store small snippets of data, and link all the snippets together in the correct order, so you can get all the file by following the chain of HTTP redirections. The first link in the chain will contain the filename, and the following links the contents of the file. This way we don’t need to keep the whole list of short URLs, only one of them will suffice! Actually, if we link the last URL in the chain back to the first, then any of the URLs will do – but now we need to be able to know which one is the first link. We do this by appending the letter “p” (plaintext) or “z” (compressed) to the nonce.

Just like TinyURL, ito.mx doesn’t really care much for the size of the URLs. However, we don’t have a preview feature here, and the API doesn’t provide a method to get the long URL back – so if we use overly long URLs we won’t be able to retrieve them later. Also, there’s a check for overly long URLs when doing a GET request (why not in the POST request? very odd!). A safe size seems to be 128 bytes of data (256 when encoded) and two bytes (four when encoded) for the nonce, but your mileage may vary – here are a couple useful links in case you want to find out more on your own. Since this leaves us with very little room for data in each URL, we’ll have to compress the data using zlib when possible.

Note that in the TinyURL example we didn’t care about collisions. If other people happened to upload the same chunk of data, we’d get the same URL back, but that wasn’t a problem – each URL identified a single piece of data. Now we have links that not only contain data, but a pointer to the next block. If anyone happens to have used that URL already, we can’t proceed. To avoid collisions, a random nonce has to be added to each code. This reduces the amount of bytes of storage per URL, though.

With this scheme we have to create the URLs in the reverse order. That’s because when we create each link, we have to know where it points to, so we already need to have created the next one – but because of the possibility of collisions we can’t simply guess the URL to the next chunk, it has to be created first to make sure it doesn’t collide.

Finally, to prevent urllib2 from getting caught in this circular chain of redirections, we’ll set a password to the URLs. This method has much more overhead than the TinyURL one, so it’s only practical for small pieces of data, but it has the advantage of requiring only one URL to identify the whole chain, and the password protection provides us with that beautiful false sense of security that only plaintext authentication can give.

Let’s see it in action. Caught by a sudden streak of nostalgia we’ll upload the The Conscience of a Hacker (3880 bytes) with the password “h4xx04“.

  $ ./itomxfs.py upload p7_0x03_Hacker\'s\ Manifesto_by_The\ Mentor.txt h4xx0r
  Uploading: http://ito.mx/z2c75-70375f307830335f4861636b65722773204d616e69666573746f5f62795f546865204d656e746f722e747874
  Created: http://ito.mx/4e08-60c6f517e2a6d43929b8f8362d36b6d3d7ba4bcad4956473613acf456293c2527397191e5fea6f8ae3ffff6174f5635fd5bff889c400
  Created: http://ito.mx/5231-e1770f38cf02c9e35250f3224a3436d43874d0dbcd3c5e96d6f2bd7b81776e8172769e68ef24df5a1424e796fa08147a2d13248b48ef8d15f7c6c7c59d9c5a98
  Created: http://ito.mx/daed-d2a4e8c87c513a96702222717b8bc1d5a115cec89392a3c2f91d976e539a43bd0e56eb3e90b998f3b73c4565d1431c6df2da05c0707fbf5af73fa7aecfaa6737
  Created: http://ito.mx/b971-d00a9a25f41d31fdad8b9e8deeb593feaf5578fc9e2aa74dd246d40ff1b5a8306383d8c57c8f7911fffcf2754cbe93d946708c191d48854564295499d5dc0c57
  Created: http://ito.mx/b323-27efb61a6d49861aea4d8ad19544ee2a61b5151125e9b05ae83185b4dcda5c491a89e911cbc0bcabca04257960b8ebffcb6339be9be0c20e24b4188c7c5dad6b
  Created: http://ito.mx/5724-bc9a362a7519975cfd484a591ef870f7b596366150a8ba19bd05f8e4f912ed009985694724947b4daea8ad9418a92f9df651e04787d4b9ac622295d2e3931548
  Created: http://ito.mx/384d-2ad99bd2a40d0e6f92f8ef423944f945e69d77530e70afa4fd64c5639503fb7807f55576027ae967e079c5a82c2cbd784d89123e1602ba5d3b4849e4aa6195e2
  Created: http://ito.mx/f946-4aaa8afaa3c2698683a8f8e3ad579d1bb555b1646050a831118272bed2bd75be2ccbed6a527843cf2ab02081f7c5894121b31c082b3a5e0af15298927418638e
  Created: http://ito.mx/c39f-316472ce1e6d45eb0a6adf3a87c48d55a968829b85350dc9a265c248ca17aa9bda157471888eb9cd8c489952c32538a93b064e04a3a76a95c85e38983c1fa122
  Created: http://ito.mx/eb21-e535bb8c5d5556a530961da49abf890afae891fab99295cba40483ab0798be9c05fb34d157b09753bec7f6dd99e11f9cddc0d1dc4f2048daf3d7c6b284a01a96
  Created: http://ito.mx/238c-c8412e8ac5f7c629dfc969596ab7801d5c1661898061b8c80cba02b01056c3c6d9fe5e129cd3ce226ba400cc12e142812c00b36d49699b0be6c851bc441db44b
  Created: http://ito.mx/0627-ff76d955075d2d26088554d7e95658b96624712d4b6cb88dde59ddd2948c5011aa00d5934b1106d0fb36f54806efc69c1510d0517447788366453d0c1ed0eb72
  Created: http://ito.mx/987b-1ac8803d7faf9820a0339000dd6498801dd43453af467eaace9c957a60050564f2247149528139072dba6995aabac951f22964ff8084435dcd79360d526b8dce
  Created: http://ito.mx/ede1-abd07ade67c7294d78ea9d8b37bbba9c2e0849145f09617dfde569cb6c424e4255186ce6878d1182dd852c0b55e4492a84078cefdd9f4521fd40180567c964f6
  Created: http://ito.mx/6c22-92667349be7cedd6c165ecb54b5a61f1a3821f8a02b750ddc2940e55bbe0865c70a028ebeb1c75d246f13b3ad9dbcac7112a17b18c00171f971c6f18891f98ce
  Created: http://ito.mx/d8f6-8a0a74140dbf73355d844df5610450286622247b18a5a6ec17b077f94911988ee236560756c2c0c1375285e97bef1ad5a01cb50eea5bee79140dca9251b822b4
  Created: http://ito.mx/8da7-0578fa9cac761e41f703adef8bab6203588c86522c8089aeb800e5117f990c38ab006aa664ab2124a0330071286e707b31f7dca556d8dcc2aba89d95338b9721
  Created: http://ito.mx/bb5e-af2d7244c78102a277068fd0f9e14861543e663295c5c52122b64a622b3a10c06b6ce2c0d6ab49a29e81956a074a811a07f668e46f784a82008cb400f20461e2
  Created: http://ito.mx/d94d-ea60a6478abaddd5790dbaf52dce0803b0eb6473591fd58c13af994667bab2f3fa26ac735223bc2937d604fd1475ef9d37dd5d9c2e34a8d3a0a96cc14f1cdd70
  Created: http://ito.mx/3dca-83db29ccede08ceb210ddb01cf9f4e1055e476b0aef14adbbae21c9edae10632ceeda8e141c35622e69983e8433e0f3988d770ae5c48f9c1bdb39dbc0d2a56a3
  Created: http://ito.mx/005e-ca76ca1cd47450c0a51be342d71b657774a9461c0a5e0f84b4d5b3dfd46869a73bb90c10ce879eb32bcae81ddfe1f64d8ad4e98e66976a7101efe2aa673e9a34
  Created: http://ito.mx/5093-e170de7871665d1c809bb34cbd8bd4aad40f91a2ebd45c938e90933286dc356c6048939ad8870dd1c125b3553dfbea2ce3cd1d690b48c729090f6fbd86443eb4
  Created: http://ito.mx/bb57-47f09bcda67a28da4fc79fe4bcb7ce8656b36d59bc5654003cfe745c3d05d3f26ae6272d9f3d7b76cb00be3cf6c0c7252267e94f6593f233bdaae9f92faf7efe
  Created: http://ito.mx/1bcb-3dadbe6bffb7336964facb724de72124a69735bd1fe017bd20b7a5e72755757af443ff2a0963eb8c717b6d7bdaab407baf63644b61703e9a99d436b2a7111fbc
  Created: http://ito.mx/b6ec-dcc7feae0dff4acab783bee670fcf2787a7975f2e5e4c5d51faaddb13f0c74a1acde7288eeaa99af2e07a60bb6d1f94dfc122b7afc757afa7ef03887ce6dbb39
  Created: http://ito.mx/9053-78daad576b6bdc4614fdae5f71f1177f88bc76086dd282094e4aa90b4e03310d8580194977a5c98e66c43cbcd1bfefb933921f496cb79035787767ae66ee3de7
  Created: http://ito.mx/z2c75-70375f307830335f4861636b65722773204d616e69666573746f5f62795f546865204d656e746f722e747874

This is how we download the file using the first link.

  $ ./itomxfs.py download http://ito.mx/z2c75-70375f307830335f4861636b65722773204d616e69666573746f5f62795f546865204d656e746f722e747874 h4xx0r
  Reading: http://ito.mx/z2c75-70375f307830335f4861636b65722773204d616e69666573746f5f62795f546865204d656e746f722e747874
  Reading: http://ito.mx/9053-78daad576b6bdc4614fdae5f71f1177f88bc76086dd282094e4aa90b4e03310d8580194977a5c98e66c43cbcd1bfefb933921f496cb79035787767ae66ee3de7
  Reading: http://ito.mx/b6ec-dcc7feae0dff4acab783bee670fcf2787a7975f2e5e4c5d51faaddb13f0c74a1acde7288eeaa99af2e07a60bb6d1f94dfc122b7afc757afa7ef03887ce6dbb39
  Reading: http://ito.mx/1bcb-3dadbe6bffb7336964facb724de72124a69735bd1fe017bd20b7a5e72755757af443ff2a0963eb8c717b6d7bdaab407baf63644b61703e9a99d436b2a7111fbc
  Reading: http://ito.mx/bb57-47f09bcda67a28da4fc79fe4bcb7ce8656b36d59bc5654003cfe745c3d05d3f26ae6272d9f3d7b76cb00be3cf6c0c7252267e94f6593f233bdaae9f92faf7efe
  Reading: http://ito.mx/5093-e170de7871665d1c809bb34cbd8bd4aad40f91a2ebd45c938e90933286dc356c6048939ad8870dd1c125b3553dfbea2ce3cd1d690b48c729090f6fbd86443eb4
  Reading: http://ito.mx/005e-ca76ca1cd47450c0a51be342d71b657774a9461c0a5e0f84b4d5b3dfd46869a73bb90c10ce879eb32bcae81ddfe1f64d8ad4e98e66976a7101efe2aa673e9a34
  Reading: http://ito.mx/3dca-83db29ccede08ceb210ddb01cf9f4e1055e476b0aef14adbbae21c9edae10632ceeda8e141c35622e69983e8433e0f3988d770ae5c48f9c1bdb39dbc0d2a56a3
  Reading: http://ito.mx/d94d-ea60a6478abaddd5790dbaf52dce0803b0eb6473591fd58c13af994667bab2f3fa26ac735223bc2937d604fd1475ef9d37dd5d9c2e34a8d3a0a96cc14f1cdd70
  Reading: http://ito.mx/bb5e-af2d7244c78102a277068fd0f9e14861543e663295c5c52122b64a622b3a10c06b6ce2c0d6ab49a29e81956a074a811a07f668e46f784a82008cb400f20461e2
  Reading: http://ito.mx/8da7-0578fa9cac761e41f703adef8bab6203588c86522c8089aeb800e5117f990c38ab006aa664ab2124a0330071286e707b31f7dca556d8dcc2aba89d95338b9721
  Reading: http://ito.mx/d8f6-8a0a74140dbf73355d844df5610450286622247b18a5a6ec17b077f94911988ee236560756c2c0c1375285e97bef1ad5a01cb50eea5bee79140dca9251b822b4
  Reading: http://ito.mx/6c22-92667349be7cedd6c165ecb54b5a61f1a3821f8a02b750ddc2940e55bbe0865c70a028ebeb1c75d246f13b3ad9dbcac7112a17b18c00171f971c6f18891f98ce
  Reading: http://ito.mx/ede1-abd07ade67c7294d78ea9d8b37bbba9c2e0849145f09617dfde569cb6c424e4255186ce6878d1182dd852c0b55e4492a84078cefdd9f4521fd40180567c964f6
  Reading: http://ito.mx/987b-1ac8803d7faf9820a0339000dd6498801dd43453af467eaace9c957a60050564f2247149528139072dba6995aabac951f22964ff8084435dcd79360d526b8dce
  Reading: http://ito.mx/0627-ff76d955075d2d26088554d7e95658b96624712d4b6cb88dde59ddd2948c5011aa00d5934b1106d0fb36f54806efc69c1510d0517447788366453d0c1ed0eb72
  Reading: http://ito.mx/238c-c8412e8ac5f7c629dfc969596ab7801d5c1661898061b8c80cba02b01056c3c6d9fe5e129cd3ce226ba400cc12e142812c00b36d49699b0be6c851bc441db44b
  Reading: http://ito.mx/eb21-e535bb8c5d5556a530961da49abf890afae891fab99295cba40483ab0798be9c05fb34d157b09753bec7f6dd99e11f9cddc0d1dc4f2048daf3d7c6b284a01a96
  Reading: http://ito.mx/c39f-316472ce1e6d45eb0a6adf3a87c48d55a968829b85350dc9a265c248ca17aa9bda157471888eb9cd8c489952c32538a93b064e04a3a76a95c85e38983c1fa122
  Reading: http://ito.mx/f946-4aaa8afaa3c2698683a8f8e3ad579d1bb555b1646050a831118272bed2bd75be2ccbed6a527843cf2ab02081f7c5894121b31c082b3a5e0af15298927418638e
  Reading: http://ito.mx/384d-2ad99bd2a40d0e6f92f8ef423944f945e69d77530e70afa4fd64c5639503fb7807f55576027ae967e079c5a82c2cbd784d89123e1602ba5d3b4849e4aa6195e2
  Reading: http://ito.mx/5724-bc9a362a7519975cfd484a591ef870f7b596366150a8ba19bd05f8e4f912ed009985694724947b4daea8ad9418a92f9df651e04787d4b9ac622295d2e3931548
  Reading: http://ito.mx/b323-27efb61a6d49861aea4d8ad19544ee2a61b5151125e9b05ae83185b4dcda5c491a89e911cbc0bcabca04257960b8ebffcb6339be9be0c20e24b4188c7c5dad6b
  Reading: http://ito.mx/b971-d00a9a25f41d31fdad8b9e8deeb593feaf5578fc9e2aa74dd246d40ff1b5a8306383d8c57c8f7911fffcf2754cbe93d946708c191d48854564295499d5dc0c57
  Reading: http://ito.mx/daed-d2a4e8c87c513a96702222717b8bc1d5a115cec89392a3c2f91d976e539a43bd0e56eb3e90b998f3b73c4565d1431c6df2da05c0707fbf5af73fa7aecfaa6737
  Reading: http://ito.mx/5231-e1770f38cf02c9e35250f3224a3436d43874d0dbcd3c5e96d6f2bd7b81776e8172769e68ef24df5a1424e796fa08147a2d13248b48ef8d15f7c6c7c59d9c5a98
  Reading: http://ito.mx/4e08-60c6f517e2a6d43929b8f8362d36b6d3d7ba4bcad4956473613acf456293c2527397191e5fea6f8ae3ffff6174f5635fd5bff889c400
  Merging 26 parts
  Writing: p7_0x03_Hacker's Manifesto_by_The Mentor.txt

Let’s see now what happens if we start downloading the file not from the first link, but from any other link in the chain.

  $ ./itomxfs.py download http://ito.mx/6c22-92667349be7cedd6c165ecb54b5a61f1a3821f8a02b750ddc2940e55bbe0865c70a028ebeb1c75d246f13b3ad9dbcac7112a17b18c00171f971c6f18891f98ce h4xx0r
  Reading: http://ito.mx/6c22-92667349be7cedd6c165ecb54b5a61f1a3821f8a02b750ddc2940e55bbe0865c70a028ebeb1c75d246f13b3ad9dbcac7112a17b18c00171f971c6f18891f98ce
  Reading: http://ito.mx/ede1-abd07ade67c7294d78ea9d8b37bbba9c2e0849145f09617dfde569cb6c424e4255186ce6878d1182dd852c0b55e4492a84078cefdd9f4521fd40180567c964f6
  Reading: http://ito.mx/987b-1ac8803d7faf9820a0339000dd6498801dd43453af467eaace9c957a60050564f2247149528139072dba6995aabac951f22964ff8084435dcd79360d526b8dce
  Reading: http://ito.mx/0627-ff76d955075d2d26088554d7e95658b96624712d4b6cb88dde59ddd2948c5011aa00d5934b1106d0fb36f54806efc69c1510d0517447788366453d0c1ed0eb72
  Reading: http://ito.mx/238c-c8412e8ac5f7c629dfc969596ab7801d5c1661898061b8c80cba02b01056c3c6d9fe5e129cd3ce226ba400cc12e142812c00b36d49699b0be6c851bc441db44b
  Reading: http://ito.mx/eb21-e535bb8c5d5556a530961da49abf890afae891fab99295cba40483ab0798be9c05fb34d157b09753bec7f6dd99e11f9cddc0d1dc4f2048daf3d7c6b284a01a96
  Reading: http://ito.mx/c39f-316472ce1e6d45eb0a6adf3a87c48d55a968829b85350dc9a265c248ca17aa9bda157471888eb9cd8c489952c32538a93b064e04a3a76a95c85e38983c1fa122
  Reading: http://ito.mx/f946-4aaa8afaa3c2698683a8f8e3ad579d1bb555b1646050a831118272bed2bd75be2ccbed6a527843cf2ab02081f7c5894121b31c082b3a5e0af15298927418638e
  Reading: http://ito.mx/384d-2ad99bd2a40d0e6f92f8ef423944f945e69d77530e70afa4fd64c5639503fb7807f55576027ae967e079c5a82c2cbd784d89123e1602ba5d3b4849e4aa6195e2
  Reading: http://ito.mx/5724-bc9a362a7519975cfd484a591ef870f7b596366150a8ba19bd05f8e4f912ed009985694724947b4daea8ad9418a92f9df651e04787d4b9ac622295d2e3931548
  Reading: http://ito.mx/b323-27efb61a6d49861aea4d8ad19544ee2a61b5151125e9b05ae83185b4dcda5c491a89e911cbc0bcabca04257960b8ebffcb6339be9be0c20e24b4188c7c5dad6b
  Reading: http://ito.mx/b971-d00a9a25f41d31fdad8b9e8deeb593feaf5578fc9e2aa74dd246d40ff1b5a8306383d8c57c8f7911fffcf2754cbe93d946708c191d48854564295499d5dc0c57
  Reading: http://ito.mx/daed-d2a4e8c87c513a96702222717b8bc1d5a115cec89392a3c2f91d976e539a43bd0e56eb3e90b998f3b73c4565d1431c6df2da05c0707fbf5af73fa7aecfaa6737
  Reading: http://ito.mx/5231-e1770f38cf02c9e35250f3224a3436d43874d0dbcd3c5e96d6f2bd7b81776e8172769e68ef24df5a1424e796fa08147a2d13248b48ef8d15f7c6c7c59d9c5a98
  Reading: http://ito.mx/4e08-60c6f517e2a6d43929b8f8362d36b6d3d7ba4bcad4956473613acf456293c2527397191e5fea6f8ae3ffff6174f5635fd5bff889c400
  Reading: http://ito.mx/z2c75-70375f307830335f4861636b65722773204d616e69666573746f5f62795f546865204d656e746f722e747874
  Reading: http://ito.mx/9053-78daad576b6bdc4614fdae5f71f1177f88bc76086dd282094e4aa90b4e03310d8580194977a5c98e66c43cbcd1bfefb933921f496cb79035787767ae66ee3de7
  Reading: http://ito.mx/b6ec-dcc7feae0dff4acab783bee670fcf2787a7975f2e5e4c5d51faaddb13f0c74a1acde7288eeaa99af2e07a60bb6d1f94dfc122b7afc757afa7ef03887ce6dbb39
  Reading: http://ito.mx/1bcb-3dadbe6bffb7336964facb724de72124a69735bd1fe017bd20b7a5e72755757af443ff2a0963eb8c717b6d7bdaab407baf63644b61703e9a99d436b2a7111fbc
  Reading: http://ito.mx/bb57-47f09bcda67a28da4fc79fe4bcb7ce8656b36d59bc5654003cfe745c3d05d3f26ae6272d9f3d7b76cb00be3cf6c0c7252267e94f6593f233bdaae9f92faf7efe
  Reading: http://ito.mx/5093-e170de7871665d1c809bb34cbd8bd4aad40f91a2ebd45c938e90933286dc356c6048939ad8870dd1c125b3553dfbea2ce3cd1d690b48c729090f6fbd86443eb4
  Reading: http://ito.mx/005e-ca76ca1cd47450c0a51be342d71b657774a9461c0a5e0f84b4d5b3dfd46869a73bb90c10ce879eb32bcae81ddfe1f64d8ad4e98e66976a7101efe2aa673e9a34
  Reading: http://ito.mx/3dca-83db29ccede08ceb210ddb01cf9f4e1055e476b0aef14adbbae21c9edae10632ceeda8e141c35622e69983e8433e0f3988d770ae5c48f9c1bdb39dbc0d2a56a3
  Reading: http://ito.mx/d94d-ea60a6478abaddd5790dbaf52dce0803b0eb6473591fd58c13af994667bab2f3fa26ac735223bc2937d604fd1475ef9d37dd5d9c2e34a8d3a0a96cc14f1cdd70
  Reading: http://ito.mx/bb5e-af2d7244c78102a277068fd0f9e14861543e663295c5c52122b64a622b3a10c06b6ce2c0d6ab49a29e81956a074a811a07f668e46f784a82008cb400f20461e2
  Reading: http://ito.mx/8da7-0578fa9cac761e41f703adef8bab6203588c86522c8089aeb800e5117f990c38ab006aa664ab2124a0330071286e707b31f7dca556d8dcc2aba89d95338b9721
  Reading: http://ito.mx/d8f6-8a0a74140dbf73355d844df5610450286622247b18a5a6ec17b077f94911988ee236560756c2c0c1375285e97bef1ad5a01cb50eea5bee79140dca9251b822b4
  Merging 26 parts
  Writing: p7_0x03_Hacker's Manifesto_by_The Mentor.txt

And that’s it for today. If you found this post interesting, you can download the tools linked right below this paragraph. If you implement more parasitic storages in URL shorteners, let me know!

Update:

• Ryan at Freebern.org has also implemented an URL shortener filesystem in PHP called Furl! Check out the Google Code project page: http://code.google.com/p/furl/.

Downloads

tinyurlfs.py

itomxfs.py

Source code

You can get the source code at Github.

Of course, this can also be used for evil purposes.

So I made myself a Python module to toy with these things. It can be used as a command line tool as well as a module to be imported in projects of your own, and contains no non-standard dependencies. You can download it from here: shorturl.py.

For example, this is how you convert a long URL into a short one (the shortest possible choice is automatically selected):

    $ ./shorturl.py http://www.example.com/
    http://u.nu/63e

And it’s reverse, converting that short URL back into the long one:

    $ ./shorturl.py -l http://u.nu/63e
    http://www.example.com/

You can also choose a specific URL shortener service. Let’s try Twitter’s new favorite, bit.ly:

    $ ./shorturl.py http://www.example.com/ -u bit.ly
    http://bit.ly/3hDSUb

Now, the fun stuff what would happen if we try to shorten, say, a javascript: link? We can find out with the -t switch (output edited for brevity):

    $ python shorturl.py -t "javascript:alert('Pwn3d');"
    Testing bit.ly:
    RuntimeError: No data returned by URL shortener API
    Testing cru.ms:
    RuntimeError: <h3>URL has wrong format! Forgot 'http://'?</h3>
    Testing easyuri.com:
            Short [1]: http://easyuri.com/657ae
            Long  [0]: http://javascript:alert('Pwn3d');
    Testing tinyurl.com:
            Short [1]: http://tinyurl.com/yhabhth
            Long  [0]: javascript:alert('Pwn3d');
    Testing xrl.us:
            Short [1]: http://xrl.us/bgsge5
            Long  [0]: http://xrl.usjavascript:alert('Pwn3d');

Most services either report an error condition or simply don’t return anything at all. Two services return a garbage response (easyuri.com and xrl.us). But to my surprise TinyURL, the second largest URL shortener service is vulnerable to this! True, this is probably not news to many of you, and blocking this kind of links doesn’t provide that much of an advantage (clicking on a link with an unknown target is pretty much game over anyway) but I still fail to see the reason to support it in the first place. I can’t think of any legitimate reasons to shorten a javascript link.

As a matter of fact, TinyURL seems to accept anything you send it, may it look like a valid URL or not. This has lead some people to build a rogue filesystem called TinyDisk, world readable, yet very hard (if not impossible) to trace back to it’s creator. The download link seems to be down but the Slashdot article is still there. I also found this article explaining how such a filesystem would work.

Back on the matter, the way people react to URL shorteners make them perfect for an attacker to hide the true target of his/her exploits, or bypass spam filters as it’s being done in the wild for quite some time now. There are some countermeasures. Many URL shorteners (including TinyURL) have a preview feature to let you examine the URL target before going there (for example, http://preview.tinyurl.com/yhabhth shows the sample javascript code we used above). The Echofon plugin for Firefox is a Twitter client that makes heavy use of URL shorteners, and among other things it automatically expands short URLs when the mouse hovers over them. The longurl.com service offers URL expansion as well, both from the web and as a Firefox plugin.

But… how well do they behave when you shorten the URL multiple times? First, let’s try shortening a link with TinyURL three successive times:

    $ ./shorturl.py -c 3 -u tinyurl.com http://www.example.com/
    http://tinyurl.com/ya3k2yy

The preview feature only seems to work for the first redirection (see for yourself). Now let’s try shortening our example URL three times, using random providers:

    $ ./shorturl.py http://www.example.com -v -c 3
    Service: thurly.net
    Service: ito.mx
    Service: cru.ms
    http://cru.ms/53c14

Just for fun, how about an infinite recursion? First, we create a link to a shortened URL that doesn’t yet exist.

    $ python shorturl.py -v -c 5 http://tinyurl.com/thisisaninfiniteloop
    Service: ito.mx
    Service: migre.me
    Service: thurly.net
    Service: is.gd
    Service: xrl.us
    Service: onodot.com
    http://onodot.com/xeaw

Now let’s go to the TinyURL webpage and create the link manually. Voila! We have an infinite redirection. The longurl.com service seems to break now (see for yourself) but this is of no consequence – the link doesn’t really take you anywhere. Other apps like wget get stuck in an infinite loop however. This is probably of no real use to an attacker but I thought it was fun to try it out.

As a final note: while writing this I came across this article describing even more risks inherent to URL shorteners: information disclosure. A dedicated attacker could dump randomly chosen shortened URLs to retrieve information like corporate intranet links, user IDs and even passwords. I liked the idea, so I whipped out a small script to test it out:

    from random import randint
    from shorturl import longurl, is_short_url

    characters = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
    while 1:
        token = ''.join([ characters[ randint(0, len(characters) - 1) ] for i in xrange(6) ])
        try:
            url = longurl('http://bit.ly/%s' % token)
        except Exception:
            print token
            continue
        if not is_short_url(url):
            print "%s => %s" % (token, url)

But the results of bruceforcing didn’t seem so great. Then again, one could also wonder how random this short URLs really are… maybe there’s a way to predict them, or at least reduce the number of URLs one has to try. One could also try a dictionary based approach for services like bit.ly, which provide the users the ability to pick custom short URLs. Update: Those TinyURL links weren’t so random after all here’s an interesting project on URL scraping.

Read Part 2: parasitic storage

Download

shorturl.py

bitlybf.py

Source code

You can get the source code at Github.