Hans5958 (dd567a56) at 19 Mar 14:32

... and 1 more commit

Hans5958 (4fe98ea0) at 13 Mar 17:15

... and 2 more commits

I realised that I should upload the URL replacer logs: $5969272

I forgot to update this when I'm finishing it, but here's a summary.

• The old laptop, which has all the data, can't be turned on anymore, and accessing the hard drive directly is not feasible in the mean time. It took a while to get all the views again.
• At that time, I have improved the script in various ways, from adding pushing to the pool directly (or the staging pool), random queue system to reduce bias, better date filtering, etc. I haven't pushed some of the improvements from the old laptop, which means I redid some of it.
• The discovery loop now includes select groups of songs (such as those with "out of scope (music PV) as it has a 100k view threshold and those with "short version" and "vertical version" tags as they have a great chance to have more than 100k views)
• The pool update/push got stalled multiple times, because stalling it means I have to redo the data collection, since new songs may be added, which means I have to stall it again, and makes a vicious loop. I finally decided to do the final collect-discover loop and set 12 March 2026 as the date of the push, which I announced a day before to review.

The pool update/push on 12 March 2026 is done successfully. Here are list showing the count before and after.

YouTube

• 100K: 9298 → 11109 (+1811)
• 1M: 3130 → 3494 (+364)
• 10M: 495 → 524 (+29)
• 100M: 15 → 15 (=)

NND

• 100K: 8196 → 8649 (+453)
• 1M: 1061 → 1091 (+30)
• 10M: 29 → 29 (=)

Bilibili

• 100K: 2129 → 3391 (+1262)
• 1M: 409 → 564 (+155)
• 10M: 23 → 23 (=)

SoundCloud

• 100K: 411 → 476 (+65)
• 1M: 31 → 40 (+9)

I've uploaded some relevant files on the attached 7z archive. This is done for posterity, especially the "most viewed" data, in case I'm unavailable and someone wanted to continue it. All of the files are shared under the same CC BY 4.0 license of VocaDB and contains...

• the last export of the pools before being updated
• the pool to be imported
• the "most viewed" data

exports.7z

Some notes:

• All view counts have been updated (if available) and the order has been adjusted (songs with no view count is sorted by ID)
• More songs are added on the list thanks to the discovery process.
• The exact view count on Bilibili is now available. Yes, yt-dlp can get it. Gone are the days of "211k" or something like that.
• For YouTube, I have to tune the extractor arguments so much to avoid getting rate limit errors. I even have to change player clients (android_sdkless → android/ios) because it took so long, and YouTube is changing the internals so much.
• For SoundCloud, I finally got a way to get the view count via a single API response. Weeks after that, they did some changes that hinders this significantly. The singular API response didn't work (I haven't investigated it why), and yt-dlp also didn't work. At least I'm not alone since VocaDB itself also have the same issues.
• For Bilibili and SoundCloud, Only "Official" PVs are considered on the pool. I kept this because it is mentioned as so on the description of the pools. There are no such descriptions on YouTube and NND, so I consider all the PVs.

Thank you for sticking around for this long, and I apologize for taking too much time for this. Pushing this to the finish line feels like I've finally paid off a mental mortgage I had for so long.

See you some time later when I'm around to do this again to refresh the data.

This dump run is done. Some important notes:

• Due to my current limitations, YAF is started earlier; before ELC is done but after ELC has been run for a while. This shouldn't affect the resulting edits, as YAF is slow (needs manual intervention compared to ELC) and ELC is quite ahead enough.

• A YAF rerun is done on 11 March for albums (singles) which has the URL of the PV added as an external link (instead of being added on the song entry). A URL replacer is done to convert the PV link to the playlist/album link before the I run YAF, which I gave details on Discord as follows:

  I wanted to make a URL replacer function to handle those that entered the YouTube song/video instead of the album/playlist. Most of this are caused by people copying TuneCore Japan URLs which links directly to the video (or song) instead of the album itself (e.g. https://vocadb.net/Al/52791 https://vocadb.net/Album/ViewVersion/278075), though the replacer could also remove duplicate URLs to handle old cases where both the song and the album is added on the entry at the same time (just added it).

  The replacer will use the shared URL processor to reverse search the album from the song (it won't remove those that are not included on albums), as well as checking if the playlist is already added, which means it can be safely removed. The videos can be added to their respective song entries using YAF (planned to do after it)

With that settled, here are the logs of the run:

• Bandcamp albums that are not in VocaDB (01/03/2026 dump)
• Booth products that are not in VocaDB (01/03/2026 dump)
• Albums with KARENT artist and without karent.jp external link (01/03/2026 dump)
• KARENT albums that are not in VocaDB (01/03/2026 dump)
• External links crawler (VocaDB): Result for full run (01/03/2026 dump)
• YouTube album filler (VocaDB): Result for full run (01/03/2026 dump)

And, that's all for this run. Any queries? Don't hesitate to contact me either here on GitLab, on the Discord server, or on VocaDB (just made a topic there, discussions welcome!). See you in the next run.

Hans5958 (63c4e2d4) at 11 Mar 04:20

Hans5958 (4c2de0d9) at 05 Mar 07:55

... and 6 more commits

Snippets for this dump run:

• Bandcamp albums that are not in TouhouDB (28/02/2026 dump)
• Booth products that are not in TouhouDB (28/02/2026 dump)
• External links crawler (TouhouDB): Result for full run (28/02/2026 dump)
• YouTube album filler (TouhouDB): Result for full run (28/02/2026 dump)

Any comments? Feel free to post it here, the TouhouDB Discord server, or my user page on TouhouDB. See you in the next run!