Tyler Hallada - Blog

Row Your Boat: How I made a boat physics simulation inside Oblivion Remastered

Sun, 24 Aug 2025 00:00:00 +0000

If creativity is borne out of constraints, creating mods for games must be one of the most creative things you can do as a programmer. It’s just so fun to hack a game engine to do something it was never supposed to do.

This blog post goes into depth describing a mod I made for the game Oblivion Remastered, a full graphics overhaul of the classic 2006 open-world RPG Elder Scrolls IV: Oblivion . The mod, Row Your Boat, adds something the game engine was never supposed to support: a useable rowboat. The player can purchase the rowboat, row it on any waterway, drag it over land, summon it anywhere, add upgrades, all with a realistic boat physics simulation I developed inside the limited built-in scripting engine the game engine provides.

All of the scripts included in the mod are on my GitHub here.

The Game Release

The Elder Scrolls series from Bethesda Softworks is probably the most modded series of games ever. One of my previous projects involved indexing and mapping hundreds of thousands of mods for Skyrim (the latest game in the series), so I know. A major reason why modding this series is so popular is because Bethesda releases the editor tools they used to create the games to the public for free. These tools lower the barrier of entry to modding and encourages complete beginners to try their hand at creating mods. I credit modding Elder Scrolls IV: Oblivion when I was a teenager with originally getting me interested in programming. The tool they released for the classic 2006 Oblivion is called the Construction Set. It gave modders the ability to create new items, quests, creatures, and modify pretty much anything else in the game engine Oblivion was built off of: Gamebryo.

When Oblivion Remastered was shadow-dropped on April 22, 2025 a lot of people doubted how moddable the remaster would be since Virtuos, the developer behind the remaster, had to create a Frankenstein-meld of Gamebryo and Unreal Engine to accomplish the remaster. There were no modding tools released by Virtuos or Bethesda. For the first time, modding an Elder Scrolls game wasn’t officially supported.

Despite the challenges, a whole modding community sprung up overnight and began poking and prodding at the game that was released to see what was possible. The forums lit up in frenzy and discord groups for sharing research developments were established. It wasn’t long before the first mods started trickling in on the new Oblivion Remastered site on Nexus Mods. At first they were simple desktop icon replacers or skip intro video mods but they started to get more complex as modders figured out how to modify this new game engine.

The introduction of Unreal Engine was both a blessing and a curse for modding. On one hand, there is an existing community around modding games built on Unreal Engine with their own tools and scripting system. But, on the other hand, it broke compatibility with pretty much every mod originally developed for classic Oblivion. It wasn’t going to be easy to port mods to the remaster. Modders would need to develop mods from scratch for the remaster.

Luckily, a lot of the modding tools that were used for the original game worked with the remaster after some tweaks. The community discovered that you could use the original Construction Set (along with the fantastic Construction Set Extender) to create plugins that would load in Oblivion Remastered. New beta versions of xEdit were released to support Oblivion Remastered (a GUI program for viewing and editing data inside mod plugins). A new version of Oblivion Script Extender for Oblivion Remastered (OBSE64) was released (adds new scripting functions through reverse-engineering of the game engine). It was starting to look like it would be possible to create some truly complex mods in the remaster.

The Idea

In my own play-through of the game, I had just purchased the Imperial City Waterfront shack as my character’s first home, and I was wandering around the waterfront right outside the shack and looking out over at the far shoreline of the Lake Rumare. I suddenly had the thought: wouldn’t it be great to row a boat over there? So much of Cyrodiil, the province Oblivion is set in, is carved by rivers and lakes, and it’s bordered on two ends by ocean. All of this space on the map is only accessible through swimming, which is the slowest and most cumbersome way to travel in the game (it’s also slow on a horse, which is the only “vehicle” in the game). Why shouldn’t the player be able to take any one of those boats that litter the shorelines and row it anywhere?

Researching the Original Boat Mod

In fact, I did remember downloading some sort of controllable boat mod for Oblivion way back in the day. A quick search on old Oblivion Nexus Mods reveals a bunch of mods that allow the player to pilot boats, so I knew it should be possible in theory. Out of all of these, Jason1s Pilotable Pirate Ship stood out to me. Impressively, the mod was published just a month after the initial release of the game in 2006. A lot of the other boat mods created later (reasonably) depended on and used OBSE functions for their functionality, but this mod used only the game’s built-in scripting language. OBSE64 for Oblivion Remastered isn’t yet far enough along yet to provide the same fancy functions those other mods used. So, I decided to look into the source files of Jason1’s mod to find out how he made a controllable boat with the same original limited scripting language I had to deal with.

Screenshot from Jason1s Pilotable Pirate Ship mod by Ioana.

What I found in the scripts was fascinating. Without any native vehicle physics system, collision detection API, or even basic vector and trigonometric math functions, Jason1 was able to create clever workarounds that exploited other systems in the game to create controllable boats.

For example, take a look at this excerpt from the scripts which is a custom implementation of the trigonometric functions using a 7-term Taylor series approximation of sine (since there is no built-in trig functions in the scripting language):

;Calculate radians
Set r to (d * 3.14159265) / 180
Set r2 to (d2 * 3.14159265) / 180

;Calculate offsets with Taylor series expansion of sine
Set x to r - ((r * r * r) / 6) + ((r * r * r * r * r) / 120) - ((r * r * r * r * r * r * r) / 5040)
Set y to r2 - ((r2 * r2 * r2) / 6) + ((r2 * r2 * r2 * r2 * r2) / 120) - ((r2 * r2 * r2 * r2 * r2 * r2 * r2) / 5040)

To detect collisions, the script positions a creature that has an invisible box mesh underneath the player that spans the whole length of the boat. Unlike the static boat mesh, this creature mesh is a moveable if something collides with it (through the Havok physics engine built into the game engine). So, if the player moves the boat forward or backward into terrain the invisible creature will get pushed upward to prevent clipping. Since you can query the Z (up-down) position of an object in the scripting language, the script can detect when the creature is getting pushed up by terrain and trigger a collision:

if (BoatStartMove == 0)
 ...
 set CollisionRef1 to player.PlaceAtMe aaCollider, 1, 0, 0
 CollisionRef1.SetPos z, -5
 CollisionRef1.SetScale 1.8
 Set BoatStartMove to 1
endif

... ; later on in the inner loop:

if (CollisionRef1 != 0)
 ;Check whether collision creature has been pushed upwards
 if (CollisionRef1.GetPos z > -5)
  Message "The ship has run aground"
  if (Reverse == 0)
   Set ReverseSpeed to (-10 * BoatSpeed)
   Set BoatSpeed to ReverseSpeed
   Set BoatTurn to 0
   Set Reverse to 1
   MyShip.PlaySound3D TRPMineExplode
  else
   Set BoatSpeed to 0
  endif
 else
  Set Reverse to 0
 endif
 CollisionRef1.SetPos x, BoatX
 CollisionRef1.SetPos y, BoatY
 If (CollisionRef1.GetPos z > 10)
  CollisionRef1.SetPos z, 10
 endif
 CollisionRef1.SetAngle z, BoatAngle
endif

The script also positions another custom invisible box mesh around the player specifically designed to keep them on the deck of the boat and ensure they don’t fall through the boat into the water (it’s called AntiGravRef in the script).

if (BoatStartMove == 0)
 set AntiGravRef to player.PlaceAtMe aaAntiGrav, 1, 0, 0
 AntiGravRef.SetScale 2.0
 AntiGravRef.SetRigidBodyMass 100
 ...
endif

... ; later on in the inner loop:

if (AntiGravRef != 0)
 AntiGravRef.SetPos x, PlayerX
 AntiGravRef.SetPos y, PlayerY
 AntiGravRef.SetPos z, PlayerZ
endif

This system isn’t perfect. The mod’s readme mentions that “Collision detection works best when colliding head-on and is at it’s worst when colliding with several objects along the side (like the Titanic and the iceberg). If the collider gets really out of whack, it may be neccessary to rock the ship off the obstacle by selecting commands from the menu repeatedly. Dropping anchor can also help, once the obstacle is cleared. In short, it is best to not run into things.” And, there’s a lot of comments on the mod complaining about the general bugginess of this system. However, I still thought it was impressive what was accomplished within the limitations of what was available in the scripting language.

My initial plan was to simply port Jason1’s mod to Oblivion Remastered. However, I quickly realized that too many things were changed in the game engine to make Jason1’s solution feasible in the remaster.

Getting Meshes to Appear in the Game

One of the biggest hurdles of the modding the remaster was just getting items you added in a plugin to actually appear in the game. The ESP files created by the Construction Set allowed you to define new objects and their position in the game, but this was only in the old Gamebryo side of the game engine. There was a disconnect between that and the Unreal side of the game engine that prevented these objects from showing up in-game.

Luckily, Godschildgaming created UE4SS TesSyncMapInjector to solve this problem and create the glue needed between the ESP added objects and Unreal Engine. Godschildgaming created this utility through developing a UE4SS plugin which allowed modifying the state of the Unreal Engine part of the game.

Any Oblivion Remastered mod that wanted to add new objects into the world would just need to add a dependency on UE4SS TesSyncMapInjector and create an INI or JSON config file that told TesSyncMapInjector what Unreal Engine model asset to use for each ESP object (referenced using their Formids).

Moving the Boat

In Jason1’s Pilotable Pirate Ship mod, the player starts moving the boat by clicking on hull or wheel of the ship (“activating”) which opens a MessageBox with options for moving forward, backward, or stopping (“drop anchor”). The first thing I did was try to replicate this with a rowboat model in the game.

There was already a rowboat form in the game that was an activator type (ACTI form). It’s used in one of the main quests where the player needs to activate a rowboat that an NPC left on the shoreline. So, I duplicated that into my own activator object and hooked up a script to it that would show a similar message box when activated.

To actually move the boat in a script, all I needed was use the SetPos function on a reference to my boat. To move the boat continuously, I put the SetPos in a GameMode block which runs on every frame and keep adding/subtracting to the current X and Y positions (obtained with GetPos).

I found out later that I was lucky in having attempted to try to move an activator object every frame first. Moving objects like activators, NPCs, creatures, containers, lights, and misc items dropped from inventory every frame works fine in Oblivion Remastered. However, trying to move a static object every frame does not work (for example, the big pirate ship objects in the game, which are not activators). It only works if I Disable and Enable the static reference every other frame before moving it again. Since disabling a reference hides the mesh from the game, this causes the static reference to appear to flicker really badly as it moves. I still haven’t found a solution to this. I think I need to modify the Unreal Engine model asset to add a component that allows it to transform every frame, but the tools to edit the assets are really limited right now and I couldn’t figure it out. Luckily, everything I needed to move in my mod wasn’t one of these problematic static meshes (though, it would be nice to someday add a controllable full pirate ship in addition to a rowboat).

I was expecting to run into the problem Jason1 ran into with players clipping through the boat mesh and falling into the water while the boat was moving, but it turns out Unreal Engine actually handles this better! It had no problems with moving the player and keeping them on the deck while the boat was moving.

Turning the Boat

The other critical part of moving the boat is turning it so the player can dictate where the boat is moving. Jason1’s mod achieved this by locking the boat angle to the player’s view angle. So for example, when the player turned to look right, the boat would follow and turn right.

This is where the trigonometry comes into play. Instead of using the Taylor series that Jason1 used to approximate the sine, cosine, and tangent functions, I opted to use this script made by Galsiah that I found on the CS Wiki which claimed to be faster and more accurate than the Taylor series.

; Script originally by Galsiah.
; See: https://cs.uesp.net/wiki/Trigonometry_Functions#Galsiah_Version
set RYB.ang to (RYB.ang * RYB.degToRad)
set RYB.n to 1
if (RYB.ang > 4.7123)
    set RYB.ang to (RYB.ang - 6.2832)
elseif (RYB.ang > 1.5708)
    set RYB.ang to (RYB.ang - 3.1416)
    set RYB.n to -1
endif
set RYB.t2 to (RYB.ang * RYB.ang)
set RYB.sin to RYB.n*(RYB.ang*(1 - RYB.t2*0.16605 + 0.00761*RYB.t2*RYB.t2))
set RYB.cos to RYB.n*(1 - RYB.t2*0.4967 + 0.03705*RYB.t2*RYB.t2)
set RYB.tan to (RYB.sin/RYB.cos)

This method employs multiple techniques like reducing the domain of the angles to [−π/2,π/2] (plus some wraparound logic) to prevent accuracy issues with large angles, efficient Horner’s method-style evaluation to reduce the number of calculations, and carefully chosen magic-number coefficients which were likely derived from minimax approximation or curve fitting in order to approximate the trigonometric functions as close as possible while also keeping the calculation fast.

With the ability to calculate sine and cosine, I could now calculate the next X and Y position of the boat given the current speed and the current angle of the boat:

set BoatX to BoatX + (sin * FrameBoatVelocity)
set BoatY to BoatY + (cos * FrameBoatVelocity)
...
BoatRef.SetPos x, BoatX
BoatRef.SetPos y, BoatY

To actually change the angle of the boat depending on the player look angle, I developed a similar solution to Jason1’s. Except, instead of locking the boat angle directly to player angle, I kept them independent and instead gradually modified the boat angle towards the player angle every frame. This made the turning feel much more natural and gave the rowboat realistic weight. The rate of turning also slows down as the boat speed slows down.

I also added a dead-zone a few degrees out from either side of the center line of the boat so if the player moves slightly it doesn’t cause the whole boat to move. This made turning the boat much more intentional and avoided the boat weaving too much side to side when the player was just attempting to go forward.

The turn rate also decays. So if the player stops turning the boat by looking directly ahead, the boat will naturally slow turning until it stops turning.

Video of turning the rowboat on the water by looking left and right

Detecting Collision

Since the meshes are handled by Unreal Engine in Oblivion Remastered, I didn’t think the same method Jason1 used in his original mod would work for the remaster. I also wanted a better collision detection system since it sounded like there was a lot of issues with the method of using an invisible collision plane below the player.

At first, I tried spawning in two objects: one at the bow of the boat and another a short distance in front of the boat. And, then using a script to make the first object fire a projectile spell towards the second object. I could use OnMagicEffectHit on the second object to listen for the spell hitting it. If it didn’t hit within some time limit then I could assume the spell collided with something else (e.g. terrain) which means the boat collided with something. So in effect: literal ray casting.

This sort of worked, but a major problem with this approach was that I couldn’t find a way to make the spell casting silent and invisible. It was really annoying to see a constant stream of particles and hear the spell casting sound effects at the front of the boat while it was moving. I tried setting the spell effect visuals to NONE in the Construction Set but this seemed to cause the spell to not work at all in Oblivion Remastered.

Another issue with that approach was that the second object that receives the spell projectile needs to be a mesh that has collision so that the spell can actually collide with it instead of passing right through. But, I couldn’t find a single object in the game that was both invisible and had collision. The original game had invisible collision boxes that blocked the player from certain areas, but these boxes didn’t seem to have collision in Oblivion Remastered.

While looking for an object that had both collision and was invisible, I tried a rat with a permanent invisibility magic effect power. When I tried this I discovered that the game will always ensure that an actor is placed above objects or terrain to avoid clipping with them if you request to move the actor within an object or below terrain. I realized that I could abuse this behavior alone to detect collision since I could try to place an invisible and immobile actor in front of the boat, wait one frame, and then query the actual Z position of the actor to see where the game actually placed them. If the Z position returned is higher than the position I requested, then I know there is something in front of the boat that should cause a collision with the boat.

It took me a surprisingly long time to find an actor in the game that could be completely invisible and completely silent. The best solution I found was creating a new NPC that has the VampireRace assigned to it. It seems like at some point in the game’s development there were plans to make vampires a different race in the game, but that was scrapped so that all races could contract vampirism instead. So, this race is not used in vanilla oblivion, and because of that there are no voice lines assigned to the race. This was important because NPC of other races would occasionally say idle dialogue lines or remark on things happening and break the illusion. I also set the scale of the NPC reference in the Construction Set to as small as the game engine would allow (like ant sized basically). I also used a bunch of other script functions to make the NPC invisible and turn off any AI that would cause it to react to the player or other actors in the area:

RYBColliderRef.SetActorAlpha 0.0
RYBColliderRef.SetActorRefraction 10.0
RYBColliderRef.AddSpell MG14JskarInvis
RYBColliderRef.SetActorValue Aggression 0
RYBColliderRef.SetActorValue Blindness 100
RYBColliderRef.ModActorValue Sneak 0
; RYBColliderRef.SetActorsAI 0 ; causes crash in 1.511.102.0
RYBColliderRef.SetDestroyed 1

Video of the boat moving forward until it collides with a beach

It’s still not a 100% perfect solution, since the vampire can make occasional splashing sounds when it hits the water which sounds quite glitchy when it happens every few frames. When I tried a slaughterfish, the splashing sound didn’t happen but I couldn’t figure out how to silence the slaughterfish’s idle sounds which were more annoying.

To handle collisions while turning the boat, the invisible vampire (I call it the “collider” in my script) is spawned in the direction of travel. So, when moving in reverse, the collider is spawned behind the boat instead of in front.

I was concerned that moving an NPC every frame would cause a big performance hit. In practice, I don’t think I noticed any real effect on my machine. But, to be safe, I found the optimal frequency for placing the collider that minimized the how often it had to move while also still detecting collisions fast enough.

So that’s how I detect collision my mod: hanging a tiny invisible, mute, dumb vampire off the front of the boat until it bashes into something 😉.

Rowing the Boat

While moving the boat automatically through the MessageBox menu was convenient, it was also cumbersome and awkward to interact with. I wanted to create a way to really make the player feel like they are rowing the boat for realism and ✨immersion✨.

Ideally the player would just hop on the boat and press some keybind to start moving forward. But, the built-in scripting language has no way to detect keypresses. OBSE64 may add this ability eventually, and mod developers have since found ways to trigger events on keypress through UE4SS scripts. At the time I made the mod, I decided to go with a spell instead.

In essence, the spell cast keybind would become the “go forward” keybind while the player was on the boat. I just had to create a new spell called “Row”, that when cast would trigger something in my script to tell it to start moving the boat forward. Oblivion was developed with custom-scripted spells in mind, so there are actually quite a few hooks built into the scripting language for magic effects. Specifically the blocktype ScriptEffectStart was very useful for this.

To allow the player to easily row backwards, I used the script function IsSneaking to change the direction of rowing if the player was sneaking when the spell was cast.

begin ScriptEffectStart
    if (Player.IsSneaking)
        set RYB.TriggerRowCast to 2
    else
        set RYB.TriggerRowCast to 1
    endif
end

The script assigned to my custom Row spell above sets a variable in my main quest script (attached to the quest named RYB) to 1 or 2. The quest script will detect the variable change and handle it by moving the boat forward or backwards respectively. Basically, this is really primitive function-calling between different scripts in the game.

To make rowing even more realistic, I also added a small Damage Fatigue on self effect to the spell so rowing slowly drained the player’s fatigue over time (just like how running does in the game). This made it so the player couldn’t row indefinitely unless they used some other spell or potion to bolster their fatigue.

Both the Damage Fatigue effect and the Script Effect magic effects in the game cause sound and particle effects to play when cast. I found this annoying and distracting. When a spell in Oblivion has multiple magic effects assigned to it, it chooses the effect with the largest magnitude. So I just needed to find another magic effect in the game I could add that didn’t have any sounds or visual effects assigned to it and give it a large magnitude so my Row spell would use the null effects.

I initially went with the Darkness effect, which is an unused magic effect in vanilla oblivion that was cut during development (this was the magic effect being used in my initial mod release video). However, I found out from bug reports users sent after I initially released the mod that this magic effect has a bug that somehow broke Oblivion Remastered’s Night Eye effect. I suppose that serves me right for using a magic effect that was labelled in the Construction Set as “DO NOT USE”. Luckily, there was another unused magic effect in the game with no sound and visuals: Lock. This one also had the bonus that it could be cast as “touch”. Normally that means the effect will only apply to actors the player is touching right in front of them, but for the purposes of my Row spell, this gave the spell the touch spell animation which – if you squinted – sort of looked like the player pushing two oars forward with both arms.

Video of rowing the boat forward by casting the Row spell

Realistic Movement

When the Row spell is cast by the player while they are on the boat, it doesn’t immediately shoot the boat forward at its maximum speed. Instead, the Row spell cast starts a timer where, for a short time period, it adds a small amount of “force” to the boat’s current velocity every game frame. This is to simulate the effect of oars pushing through the water. After constantly rowing for a while, velocity will accumulate until the boat reaches its maximum velocity.

Since this is an effect that applies every frame, I needed to account for players having different frame rates or variations in the frame rate. It wouldn’t make any sense if the boat was faster in lower graphics settings, or slower if the player entered an area with a lower frame rate. Infamously, Oblivion has this issue with its Havok physics engine. It’s tied to the game’s frame rate which often causes bugs like objects erratically flying off shelves when the player enters an interior under high frame-rates.

To fix this, all values in my mod applied every frame are smoothed over with a value I call SmoothedDeltaTime (borrowed term from Unity’s similar value).

set SecondsPassed to GetSecondsPassed
; Clamp extreme values
if (SecondsPassed < 0.001)
    set SecondsPassed to 0.001
elseif (SecondsPassed > 0.1)
    set SecondsPassed to 0.1
endif

; Exponentially smooth the delta time to adjust for frame rate changes
set SmoothedDeltaTime to ((1.0 - DeltaSmoothingFactor) * SmoothedDeltaTime) + (DeltaSmoothingFactor * SecondsPassed)

GetSecondsPassed returns the amount of time that has passed since the last frame.

This will smooth out frame-rate hitches so that the boat will not unexpectedly jerk forward if the player enters an area where assets are loading in and the frame rate dips low temporarily.

When the player stops rowing, the boat shouldn’t immediately stop moving. Instead the boat velocity gradually “decays” to 0 when no force is being applied. To make the decay more natural, I use exponential decay. However, since the script engine doesn’t have math functions like the natural logarithm or exponential functions, I had to approximate it with a constant value that I pre-computed.

; Speed values assume a frame rate of 60fps so readjust speed values to current frame rate
set FrameRowForce to BaseRowForce * (SmoothedDeltaTime / TargetDeltaTime)

if (Rowing == 0 && AutoRowing == 0 && BoatMoving == 2)
    ; Boat is moving, but not rowing. Decay the velocity using exponential decay.
    ; velocity = velocity * (1 - decay * time)
    if (BaseBoatVelocity != 0)
        ; Calculate decay factor based on smoothed delta time
        ; Convert to time-based decay
        ; retention^(frames) = retention^(time/frameTime)
        ; We need to calculate r^(SmoothedDeltaTime/TargetDeltaTime)

        ; Approximation since we can't do pow():
        ; For small time steps, r^t ≈ 1 + t*ln(r)
        ; ln(0.98) ≈ -0.0202
        set FrameVelocityDecay to 1 + (SmoothedDeltaTime / TargetDeltaTime) * (VelocityDecayLnRetentionFactor)
        set BaseBoatVelocity to BaseBoatVelocity * FrameVelocityDecay

     ; Stop when very slow
        if (BaseBoatVelocity > -0.2 && BaseBoatVelocity < 0.2)
            set BaseBoatVelocity to 0
            set BoatMoving to 0

Technically, my quest script doesn’t run every game frame though. The special quest variable fQuestDelayTime configures how often the script is run while the game is running. To save CPU resources, I try to keep this to a high value when the player is not near the boat, but once they start moving the boat I ramp it down to a value that would run the script at roughly 60 times per second.

Dragging the Boat

The largest and most prominent river in the game: the Niben River is actually blocked by the city Layawiin in the game, which makes it impossible to row the boat all the way on the river that goes from the Imperial City into the Topal Bay at the bottom of the map. I knew I would need to develop an alternative way to move the boat for this reason when I set out to make this mod.

I thought it would be really cool if the player could hop out of the boat and then physically drag it over land. This would allow the player to drag it over the small strip of land blocking the river next to Layawiin and continue on rowing on the other side.

After perfecting the movement of the boat over water, I already had the necessary code to move the boat over land. I would just needed to tweak it to make it feel natural.

At this point in the project, I was heavily using Claude to help me out with the script. To be honest, as a non-game developer, a lot of the 3D math involved in this project was starting to get a bit over my head. But, Claude was an amazing tool at breaking it down for me in a way I could understand and served as a great super-powered rubber duck for debugging issues.

At some point while developing the boat dragging code with Claude, I had the great idea to suggest it create an artifact by converting the OBScript code I was working on to the equivalent in JavaScript and display an interactable 2D visualization of the dragging simulation on a HTML canvas. This was super helpful in debugging a ton of issues with the dragging code because it tightened the feedback loop between making a change and then testing it out to see if it worked in the visualization. I spent a lot of time waiting to Oblivion Remastered to start up and load saves while working on this mod, so this was huge. I also told Claude to include lots of sliders for all the different variables in the dragging simulation so I could quickly tweak with them within the visualization and get the feel of the dragging really refined without even needing to load up the game.

And, now that I have this artifact, it serves as great documentation for how the dragging code works! Try it out for yourself here.

I will certainly be using LLMs to create visualizations of tricky simulations in the future. This is the sort of thing where I think AI could truly help 10x the speed and quality of code projects. To do this in the pre-LLMs days would have taken hours. Enough time that it just wouldn’t have felt worth it. But now that I can have an LLM spit it out in seconds, it would be dumb not to do it and reap the benefits of it.

The dragging code tries to simulate the player dragging the boat as if they were pulling a rope attached to the center of the boat. This allows the player to walk freely around the boat without it moving as long as they don’t make the rope taut by walking more than the rope’s length away from the center of the boat (the white circle in the visualization). Once they do, it will pull the boat with a force relative to how far away the player moved. The boat itself has friction with the ground which moderates this effect, since I wanted the dragging effect to feel slow and less practical than rowing it on water.

The boat also turns to face the bow towards the player. This makes it appear like the player is dragging the boat from its bow. The turning effect works very similarly to how the turning works on water with gradual ramp up and decay when the angle of difference enters the deadzone.

One thing the visualization does not show is how the boat behaves when dragged up or down hills (since it is only a 2D visualization). I wanted the pitch of the boat to change so that when the player drags it up a hill it pitches up to follow the slope of the terrain, and when they drag it downhill it would pitch down. Otherwise, the boat stuck out awkwardly horizontally from the side of hills while you were dragging it. It just looked unrealistic.

Since the boat doesn’t actually have any collision detection with the ground, it was quite a challenge to find a way to adjust the pitch to follow the terrain. I ultimately ended up using the player to detect the slope of the land. Since the player is pulling the boat, the boat follows the player’s path. I can query the player’s Z position to get the height of the terrain. While dragging, every 50 units or so of distance covered, the script records the current terrain height using the player Z position. Then 50 units later, it will compare the current terrain height to the height 50 units prior and use the difference to calculate the slope of the terrain. Using this angle, the boat will gradually pitch in the direction of this angle until it roughly matches the terrain’s slope. To avoid the boat clipping inside the terrain or floating too high off the terrain, the boat’s Z position is also raised or lowered relative to the terrain’s slope.

The effect isn’t perfect, but it’s surprising how much of a difference it makes. It’s certainly close enough to make a convincing illusion that the boat is being dragged over land.

While dragging the boat, the player gains 200 pounds of encumbrance. This is to make the dragging more realistic since they shouldn’t be able to easily go off fighting monsters while shouldering an entire rowboat around the whole time. But, they could feasibly achieve that if they really wanted to and had the right Feather spell, potion, or enchanted equipment.

The encumbrance is achieved by adding a special “Rowboat” item to the player’s inventory. The item is scripted so if it is dropped from the player’s inventory then the dragging stops. It also has the same rowboat model assigned to it through UE4SS TesSyncMapInjector so it even looks like a rowboat in the player’s inventory preview.

Summoning the Boat

One of the first mods I downloaded for Oblivion Remastered was PushTheWinButton’s excellent Horse Whistle - Summon and Follow . There’s a reason pretty much every game these days that has horse mounts includes some sort of “whistle” mechanic that allows the player to summon their horse to their position immediately. While not exactly realistic, it’s just one of those things that smooths over gameplay so it’s not such a chore just to get playing.

For the rowboat to be a proper player vehicle, I knew I would also need to include some sort of summon ability. I implemented this with the same Row spell that is used to push the rowboat forward and backwards. If the player casts it while far enough away from the rowboat that they couldn’t feasibly be on it, then a MessageBox pops up instead. This message box includes two options: “Summon Boat” and “Place Boat Right Here”.

The difference between these two options is that “Summon Boat” tries to place the boat in somewhere in front of the player that obeys the terrain and objects around the player, whereas “Place Boat Right Here” ignores all of that and just places the boat exactly in front of the player even if it would clip with terrain or objects.

“Place Boat Right Here” was easy to implement since I just needed to use the same sin function I use for boat movement to find a spot in front of where the player is looking and then use MoveTo and SetPos to move the boat to that spot.

“Summon Boat” was a bit more complicated since I needed to first scout out a good position by spawning in the same tiny vampire collider that I use for collision detection in front of the player, wait a frame, then query its position to get the final spot the game engine decided doesn’t clip with terrain or objects, then move the collider away and spawn in the boat. This worked in most cases, but there were a few spots I found in the world where the game decides to place the collider somewhere deep underground. So that’s why I kept the “Place Boat Right Here” as a back-up if that ever happens.

I also found that I needed to disable the boat and then re-enable it after moving it, otherwise sometimes the boat would weirdly not have any collision so the player could walk right through it and it would not be activatable.

Rocking the Boat

Inspired by the classic Oblivion mod QQuix - Rock rock rock your ship, I wanted to add even more realism to the mod by adding a gentle rocking animation to the boat while it is in the water.

I achieved this (with a lot of help from Claude 🤖) by creating a system of combining three separate random sine wave oscillations:

Primary wave: 30 degrees/second
Secondary wave: 45 degrees/second
Tertiary wave: 25 degrees/second

Combining three random waves instead of a single makes the rocking more complex and unpredictable compared a single wave which would have resulted in monotonous, predictable motion.

The boat pitches by combining the primary and secondary waves:

set TargetRockPitchOffset to RockAmplitudePitch * (rockSin * 0.8 + rockSin2 * 0.2)

And rolls side-to-side by using a cosine function to create a 90-degree phase offset off the secondary and tertiary waves:

set TargetRockRollOffset to RockAmplitudeRoll * (rockCos3 * 0.7 + rockSin2 * 0.3)

And bobs up and down slightly by combining the primary and secondary waves:

set TargetRockZOffset to RockAmplitudeZ * (rockSin * 0.7 + rockSin2 * 0.3 + RockRandomPhase)

This all combines to create a fairly convincing boat rocking motion in the water:

Video of rowboat gently rocking in the water

To increase the realism, the rocking motion gets amplified by how fast the boat is moving:

; Increase rocking amplitude based on speed
set TargetRockZOffset to TargetRockZOffset * (1 + (AbsoluteBoatVelocity / BoatMaxVelocity) * RockSpeedFactor)
set TargetRockPitchOffset to TargetRockPitchOffset * (1 + (AbsoluteBoatVelocity / BoatMaxVelocity) * RockSpeedFactor * 1.5)
set TargetRockRollOffset to TargetRockRollOffset * (1 + (AbsoluteBoatVelocity / BoatMaxVelocity) * RockSpeedFactor * 0.7)

And the rocking motion also gets amplified by bad weather by querying the wind speed with GetWindSpeed and adjusting the motion accordingly:

; Apply weather factor (based on wind speed)
set WindSpeed to GetWindSpeed
set TargetRockZOffset to TargetRockZOffset * (1 + (WindSpeed * RockWeatherFactor))
; Limit extreme Z offsets that would mess with boat-in-water detection
if (BoatZ + TargetRockZOffset < -RockMaxAbsoluteZ)
    set TargetRockZOffset to -RockMaxAbsoluteZ - BoatZ
elseif (BoatZ + TargetRockZOffset > RockMaxAbsoluteZ)
    set TargetRockZOffset to RockMaxAbsoluteZ - BoatZ
endif
set TargetRockPitchOffset to TargetRockPitchOffset * (1 + (WindSpeed * RockWeatherFactor))
set TargetRockRollOffset to TargetRockRollOffset * (1 + (WindSpeed * RockWeatherFactor))

To make the motion even more responsive, I integrated the player’s weight into the rocking motion.

Video of player running back and forth along the boat and the boat pitching and rolling in response

When the player is within a 3D boundary area above the deck of the boat, I translate the player’s position into the boat’s local coordinate system:

; Calculate player position relative to boat center
set PlayerRelativeX to PlayerX - BoatX
set PlayerRelativeY to PlayerY - BoatY

; Calculate boat's forward and right vectors using BoatAngle directly
; Forward vector (bow direction): sin(BoatAngle), cos(BoatAngle)
; Right vector (starboard direction): cos(BoatAngle), -sin(BoatAngle)
; PlayerLocalY: positive = toward bow, negative = toward stern
set PlayerLocalY to PlayerRelativeX * sin + PlayerRelativeY * cos
; PlayerLocalX: positive = toward starboard, negative = toward port
set PlayerLocalX to PlayerRelativeX * cos - PlayerRelativeY * sin
set PlayerLocalZ to PlayerZ - BoatZWithRock

Then I calculate a weight effect to apply to the pitch and roll that diminishes with the distance the player is from the center of the boat. I would normally use a square root function to calculate the player’s distance from the center of the boat, but since Oblivion’s scripting language doesn’t include a square root function, I had to use Newton’s method to find the approximate square root in two iterations:

; Calculate distance from boat center for falloff effect
set PlayerDistanceFromCenter to PlayerRelativeX * PlayerRelativeX + PlayerRelativeY * PlayerRelativeY
; Newton's method square root approximation (2 iterations)
set PlayerDistanceFromCenter to PlayerWeightMaxDistanceForward ; Initial guess
set PlayerDistanceFromCenter to (PlayerDistanceFromCenter + ((PlayerRelativeX * PlayerRelativeX + PlayerRelativeY * PlayerRelativeY) / PlayerDistanceFromCenter)) / 2
set PlayerDistanceFromCenter to (PlayerDistanceFromCenter + ((PlayerRelativeX * PlayerRelativeX + PlayerRelativeY * PlayerRelativeY) / PlayerDistanceFromCenter)) / 2

Using the distance from center I can then calculate an influence factor to apply to the pitch and roll on top of the randomized environmental pitch and roll (by adding these offsets):

; Calculate influence factor (1.0 at center, 0.0 at max distance)
; Limit effect area to a box approximately the area above the boat up 60 units (PlayerWeightMaxDistanceVertical)
if (PlayerDistanceFromCenter <= PlayerWeightMaxDistanceForward && PlayerLocalZ < PlayerWeightMaxDistanceVertical && PlayerLocalX < PlayerWeightMaxDistanceSide && PlayerLocalX > -PlayerWeightMaxDistanceSide)
    set PlayerWeightInfluence to 1.0 - (PlayerDistanceFromCenter / PlayerWeightMaxDistanceForward)
else
    set PlayerWeightInfluence to 0
endif

; Calculate target pitch offset based on player's fore/aft position
; Positive PlayerLocalY = toward bow (front) = boat should pitch forward (bow dips down)
; Negative PlayerLocalY = toward stern (back) = boat should pitch backward (stern dips down)
set TargetPlayerWeightPitchOffset to PlayerLocalY * PlayerWeightPitchFactor * PlayerWeightInfluence

; Calculate target roll offset based on player's port/starboard position
; Positive PlayerLocalX = toward starboard (right) = boat should roll starboard (startboard dips down)
; Negative PlayerLocalX = toward port (left) = boat should roll port (port dips down)
set TargetPlayerWeightRollOffset to -PlayerLocalX * PlayerWeightRollFactor * PlayerWeightInfluence

; Smooth the transition to prevent jarring movements
set PlayerWeightPitchOffset to PlayerWeightPitchOffset + ((TargetPlayerWeightPitchOffset - PlayerWeightPitchOffset) * PlayerWeightSmoothingFactor * SmoothedDeltaTime / TargetDeltaTime)
set PlayerWeightRollOffset to PlayerWeightRollOffset + ((TargetPlayerWeightRollOffset - PlayerWeightRollOffset) * PlayerWeightSmoothingFactor * SmoothedDeltaTime / TargetDeltaTime)

All of the rocking motion stops if the boat collides with land or if the player moves far enough away from the boat to save unnecessary processing. For players that want to minimize the performance impact, I also added a setting in the MessageBox menu to turn off the rocking animation.

Boat Upgrades

The rowboat itself is purchasable from Sergius Verus at the Three Brothers Trade Goods in the Market District of the Imperial City. This was implemented similarly to how buying houses in the game works. You purchase a deed document from the trader which has a script attached to it with an OnAdd block that triggers when it is added to the player’s inventory which then changes the owner of the house to the player and gives them the key. In the case of my rowboat, it just flips a variable in my script which makes the rowboat operable by the player and removes the for-sale sign next to the boat where it is docked in the Waterfront District.

I thought it would be fun to also add purchasable upgrades to the rowboat that improve the experience; just like how you can purchase furniture upgrades to your house from traders. The upgrades I came up with for my rowboat are: a storage chest, a lamp, and a rope ladder that auto-deploys if the player falls overboard. In addition to these purchasable upgrades, I also included a free seat on the boat. The seat is just a standard stool in the game that I positioned so that it clips into the lip on the stern of the boat model.

Implementing these “attachments” for the boat ended up being one of the most challenging parts of developing this mod. The game sees them all as separate references. There’s no way to group them together with the rowboat as a single entity for the purposes of moving them in concert. I had to manually calculate and position each of the attachments relative to the boat’s current position.

I spent wayyy too long fiddling with the positioning of these attachments relative to the boat position. After I added the rocking animation to the boat, it got even more complicated since I had to account for the pitch and roll of the boat to determine the position of the attachments. I kept thinking I got it right, but all the attachments kept being slightly off of where they should be. I applied an extreme pitch and roll to the boat, which exaggerated the error while debugging this:

Eventually I realized that the order that yaw, roll, and pitch were applied mattered. And confusingly, the order that these should be applied can vary between game engines. I had no idea which order Oblivion Remastered used, so I had to apply them in different orders until I finally found the correct order (Oblivion ended up being a ZXY kind of engine). Here’s the final code for calculating the position of the seat at the back of the boat:

; yaw
set RYB.TempX to RYB.SeatSideOffset * RYB.cos + RYB.SeatForwardOffset * RYB.sin
set RYB.TempY to RYB.SeatForwardOffset * RYB.cos - RYB.SeatSideOffset * RYB.sin
set RYB.TempZ to RYB.SeatZOffset
; roll
set RYB.OrigX to RYB.TempX
set RYB.OrigZ to RYB.TempZ
set RYB.TempX to RYB.OrigX * RYB.CosRoll - RYB.OrigZ * RYB.SinRoll
set RYB.TempZ to RYB.OrigX * RYB.SinRoll + RYB.OrigZ * RYB.CosRoll
; pitch
set RYB.OrigY to RYB.TempY
set RYB.OrigZ to RYB.TempZ
set RYB.TempY to RYB.OrigY * RYB.CosPitch + RYB.OrigZ * RYB.SinPitch
set RYB.TempZ to -RYB.OrigY * RYB.SinPitch + RYB.OrigZ * RYB.CosPitch
; to world coords
set RYB.SeatX to RYB.BoatX + RYB.TempX
set RYB.SeatY to RYB.BoatY + RYB.TempY
set RYB.SeatZ to RYB.BoatZ + RYB.TempZ + RYB.RockZOffset

Mod Release

The finally released the mod on June 4th, 2025 on Nexus Mods and made a post on the r/oblivionmods subreddit. It was fun seeing the response. A lot of people were excited to see a mod of this complexity released. I think they saw it as a sign that Oblivion Remastered was more mod-friendly than the doubters believed, and we would all see more sophisticated mods coming out for Oblivion Remastered soon. Rock Paper Shotgun even featured my mod, which was cool!

The Mysterious Case of The Spontaneously Duplicating Rowboats

After release, I made a few updated versions that fixed various bugs that were reported by the community in the bug tracker. But, one bug that was really stumping me was the issue where players would report that sometimes their boat would spontaneously duplicate itself rendering both boats broken and unusable.

After a long session of digging through my scripts for any culprit code, I became convinced that this wasn’t an issue with my mod. I was beginning to think I was actually running into a bug of either one of my dependencies (like TesSyncMapInjector) or a bug in the game engine itself.

When I finally found a way to reproduce the issue in-game, I opened the console and selected both copies of the boat. The console reported that they had the same reference form id. Everything I knew about the original Oblivion game engine told me that this should be impossible. It seemed like the Unreal Engine side of the game engine had mistakenly duplicated the boat reference and this violation was breaking all of my scripts which were all built on the assumption that only one in-game object could have the same reference id.

Since this issue seemed to be on the Unreal Engine side of the game engine, I had no choice but to dive into the world of UE4SS lua scripting which would give me access to fiddle around with the internals of the Unreal Engine part of the game.

The full UE4SS script I wrote to fix the duplication issue is here. Essentially, I used the NotifyOnNewObject to constantly keep track of new objects getting created in Unreal Engine. If any of the items matches the class for my rowboat or any of its attachments, then I immediately delete the duplicate copies. In my testing this seemed to work pretty well, and in most cases the cleanup happened so seamlessly the player would likely not notice it happening. With the duplicates deleted, my OBScript scripts continued to operate correctly.

I never truly found out the root-cause of the duplicating boats. The process I followed to consistently reproduce the bug involved:

Move the boat outside the original cell it was placed in (outside the Imperial City Waterfront District).
Leave the boat in the new cell and then move the player to a new different cell far away from the boat.
Come back to the boat and row it back to the original cell it was placed in by the shack in the Waterfront District.

So, I suspect it has something to do with Unreal Engine getting Construction Set placed references mixed up with references that have been moved by scripts outside their originally placed cell, and somehow duplicating the reference in the process.

I haven’t gotten any reports from users that the boat duplication bug is still happening after I released a new version with the UE4SS script. I still get the occasional user reporting crashes that happen, but it’s hard to prove what mod in their load order is really causing the crash, and many users report my mod because they see the log messages my script writes in their UE4SS logs. Personally, I didn’t experience any crashes with a bare-bones load order with just my mod and its dependencies installed.

Future Work

Unless I get infatuated with Oblivion modding again, I don’t think I’ll be adding anything more to the mod anytime soon. But, if I were to, I think there’s a lot more I could add to improve the mod:

Add a controllable pirate ship with explorable interiors
Add a sailing mechanic like Valheim’s sailing where you have to constantly adjust the sails to favor the current winds
Allow hiring ship crew to help you out on the deck
Add durability to the boats so crashing full-speed into a rock has consequences
- Allow players to spend resources to repair the boat or pay someone to repair it for them
Add animated oars that move through the water as the player rows
Add rowing sounds that play when the player is rowing
Add proper follower support by giving them a place to sit or stand on the boat while it’s moving
Boat crafting
Fix the boat floating above water in some interior/city worldspace cells
- This would require a way to query the water height of the current cell. Hopefully OBSE64 eventually adds this 🤞.
(probably a different mod) add a fishing mechanic so you can fish off the boat
(probably a different mod) add a cart mod
- The dragging mechanic I added for this mod I think could be applied to carts on land. E.g. attaching a cart to your horse to get extra carrying capacity

Modmapper: Putting every Skyrim mod on a map with Rust

Wed, 05 Oct 2022 00:00:00 +0000

Modmapper is a website that I made that puts every mod for the game Elder Scrolls V: Skyrim uploaded to Nexus Mods on an interactive map.

You can view the map at https://modmapper.com.

Released in 2011, Skyrim is over a decade old now. But, its vast modding community has kept it alive and relevant to this day. Skyrim is still in the top 50 games being played on Steam in 2022 and I think it’s no coincidence that it’s also one of the most modded games ever.

The enormous and enduring modding community around the Elder Scrolls games is why I have a special fondness for the series. I was 13 when I first got interested in programming through making mods for Elder Scrolls IV: Oblivion. I quickly realized I got way more satisfaction out of modding the game than actually playing it. I was addicted to being able to create whatever my mind imagined in my favorite game.

I was working on mod for Skyrim earlier in the year¹ and was looking for the best places to put new buildings in the game world. I really wanted areas of the game world off the beaten (heavily-modded) path. After over a decade of modifications, there could be conflicts with hundreds of mods in any area I chose which could cause issues like multiple buildings overlapping or terrain changes causing floating rocks and trees.

Example of a conflict between two mods that both chose the same spot to put a lamp post and sign post so they are clipping. Screenshot by AndreySG.

Conflict between a building and rock. Screenshot by LewdManoSaurus.

Conflict between a building and a tree. Screenshot by Janquel.

Conflict between two woodcutting mills. Screenshot by Janquel.

Mod authors usually use a tool like TES5Edit to analyze a group of mod plugins to find conflicts and create patches to resolve them on a case-by-case basis. But, I was unsatisfied with that. I wanted to be assured that there would be no conflicts, or at least know the set of all possible mods out there that could conflict so I could manually patch those few mods. There was no good solution for finding conflicts across all mods though. Mod authors would need to download every Skyrim mod ever and no one has time to download all 85,000+ Skyrim mods, and no one has the computer memory to load all of those in TES5Edit at the same time.

Through that frustration, Modmapper was born with the mission to create a database of all Skyrim mod exterior cell edits. With that database I can power the website which visualizes how popular cells are in aggregate as well as allow the user to drill down to individual cells, mods, or plugins to find potential conflicts without ever having to download files themselves.

When I released the website about 7 months ago it made a big splash in the Skyrim modding community. No one had ever visualized mods on a map like this before, and it gave everyone a new perspective on the vast library of Skyrim mods. It was even featured on the front page of PC Gamer’s website. Thirteen-year-old me, who regularly read the monthly PC Gamer magazine, would have been astounded.

The comments posted to the initial mod I posted on Nexus Mods² for the project were very amusing. It seemed to be blowing their minds:

“Quite possibly this could be the best mod for Skyrim. This hands-down makes everyone’s life easier to be able to see which of their mods might be conflicting.” – Nexus Mods comment by lorddonk

“The 8th wonder of Skyrim. That’s a Titan’s work requiring a monk’s perserverance. Finally, a place to go check (in)compatibilities !!! Voted. Endorsed.” – Nexus Mods comment by jfjb2005

“They shall sing songs of your greatness! Wow, just wow.” – Nexus Mods comment by LumenMystic

“Holy Batman Tits! Be honest….. You’re a Govt Agent and made this mod during your “Terrorist Watch Shift” using a CIA super computer..” – Nexus Mods comment by toddrizzle

“What drugs are you on and can I have some?” – Nexus Mods comment by thappysnek

“This is madness! Author are some kind of overhuman?! GREAT work!”– Nexus Mods comment by TeodorWild

“You are an absolute legend. Bards will sing tales of your exploits” – Nexus Mods comment by burntwater

“I wanted to say something, but I’ll just kneel before thee and worship. This would have taken me a lifetime. Amazing.” – Nexus Mods comment by BlueGunk

“Finally found the real dragonborn” – Nexus Mods comment by yag1z

“he is the messiah!” – Nexus Mods comment by Cursedobjects

“A god amongst men.” – Nexus Mods comment by TheMotherRobbit

Apparently knowing how to program is now a god-like ability! This is the type of feedback most programmers aspire to get from their users. I knew the tool was neat and fun to build, but I didn’t realize it was that sorely needed by the community.

Today, Modmapper has a sustained user-base of around 7.5k unique visitors a month³ and I still see it mentioned in reddit threads or discord servers whenever someone is asking about the places a mod edits or what mods might be conflicting in a particular cell.

The rest of this blog post will delve into how I built the website and how I gathered all of the data necessary to display the visualization.

Downloading ALL THE MODS!

In order for the project to work I needed to collect all the Skyrim mod plugin files.

While there are a number of places people upload Skyrim mods, Nexus Mods is conveniently the most popular and has the vast majority of mods. So, I would only need to deal with this one source. Luckily, they have a nice API handy.

modmapper is the project I created to do this. It is a Rust binary that:

Uses reqwest to make requests to Nexus Mods for pages of last updated mods.
Uses scraper to scrape the HTML for individual mod metadata (since the Nexus API doesn’t provide an endpoint to list mods).
Makes requests to the Nexus Mods API to get file and download information for each mod, using serde to parse the JSON responses.
Requests the content preview data for each file and walks through the list of files in the archive looking for a Skyrim plugin file (.esp, .esm, or .esl).
If it finds a plugin, it decides to download the mod. It hits the download API to get a download link and downloads the mod file archive.
Then it extracts the archive using one of: compress_tools, unrar, or 7zip via std::process::Commmand (depending on what type of archive it is).
With the ESP files (Elder Scrolls Plugin files) extracted, I then use my skyrim-cell-dump library (more on that later!) to extract all of the cell edits into structured data.
Uses seahash to create a fast unique hash for plugin files.
It then saves all of this data to a postgres database using the sqlx crate.
Uses extensive logging with the tracing crate so I can monitor the output and have a history of a run to debug later if I discover an issue.

It is designed to be run as a nightly cron job which downloads mods that have updated on Nexus Mods since the last run.

To keep costs for this project low, I decided to make the website entirely static. So, instead of creating an API server that would have to be constantly running to serve requests from the website by making queries directly to the database, I would dump all of the data that the website needed from the database to JSON files, then upload those files to Amazon S3 and serve them through the Cloudflare CDN which has servers all over the world.

So, for example, every mod in the database has a JSON file uploaded to https://mods.modmapper.com/skyrimspecialedition/<nexus_mod_id>.json and the website frontend will fetch that file when a user clicks a link to that mod in the UI.

The cost for S3 is pretty reasonable to me ($~3.5/month), and Cloudflare has a generous free tier that allows me to host everything through it for free.

The server that I actually run modmapper on to download all of the mods is a server I already have at home that I also use for other purposes. The output of each run is uploaded to S3, and I also make a full backup of the database and plugin files to Dropbox.

A lot of people thought it was insane that I downloaded every mod⁴, but in reality it wasn’t too bad once I got all the issues resolved in modmapper. I just let it run in the background all day and it would chug through the list of mods one-by-one. Most of the time ended up being spent waiting while the Nexus Mod’s API hourly rate limit was reached on my account.⁵

As a result of this project I believe I now have the most complete set of all Skyrim plugins to date (extracted plugins only without other textures, models, etc.)⁶. Compressed, it totals around 99 GB, uncompressed: 191 GB.

After I downloaded Skyrim Classic mods in addition to Skyrim Special Edition, here are some counts from the database:

Mods	113,028
Files	330,487
Plugins	534,831
Plugin Cell Edits	33,464,556

Parsing Skyrim plugin files

The Skyrim game engine has a concept of worldspaces which are exterior areas where the player can travel to. The biggest of these being, of course, Skyrim itself (which, in the lore, is a province of the continent of Tamriel on the planet Nirn). Worldspaces are recorded in a plugin file as WRLD records.

Worldspaces are then chunked up into a square grid of cells. The Skyrim worldspace consists of a little over 11,000 square cells. Mods that make a changes to the game world have a record in the plugin (a CELL record) with the cell’s X and Y coordinates and a list changes in that cell.

There is some prior art (esplugin, TES5Edit, zedit) of open-source programs that could parse Skyrim plugins and extract this data. However, all of these were too broad for my purpose or relied on the assumption of being run in the context of a load order where the master files of a plugin would also be available. I wanted a program that could take a single plugin in isolation and skip through all of the non-relevant parts of it and dump just the CELL and WRLD record data plus some metadata about the plugin from the header as fast as possible.

After discovering the wonderful documentation on the UESP wiki about the Skyrim mod file format, I realized this would be something that would be possible to make myself. skyrim-cell-dump is a Rust library/CLI program that accepts a Skyrim mod file and spits out the header metadata of the plugin, the worlds edited/created, and all of the cells it edits/creates.

Under the hood, it uses the nom crate to read through the plugin until it finds the relevant records, then uses flate2 to decompress any compressed record data, and finally outputs the extracted data formatted to JSON with serde.

Overall, I was pretty happy with this toolkit of tools and was able to quickly get the data I needed from plugins. My only gripe was that I never quite figured out how to properly do error handling with nom. If there was ever an error, I didn’t get much data in the error about what failed besides what function it failed in. I often had to resort to peppering in a dozen dbg!() statements to figure out what went wrong.

I built it as both a library and binary crate so that I could import it in other libraries and get the extracted data directly as Rust structs without needing to go through JSON. I’ll go more into why this was useful later.

Building the website

Since I wanted to keep server costs low and wanted the site to be as fast as possible for users, I decided pretty early on that the site would be purely static HTML and JavaScript with no backend server. I decided to use the Next.js web framework with TypeScript since it was what I was familiar with using in my day job. While it does have server-side rendering support which would require running a backend Node.js server, it also supports a limited feature-set that can be exported as static HTML.

I host the site on Cloudflare pages which is available on their free tier and made deploying from Github commits a breeze⁷. The web code is in my modmapper-web repo.

The most prominent feature of the website is the interactive satellite map of Skyrim. Two essential resources made this map possible: the map tile images from the UESP skyrim map and Mapbox.

Mapbox provides a JS library for its WebGL map which allows specifying a raster tile source⁸.

The UESP team painstakingly loaded every cell in the Skyrim worldspace in the Creation Kit and took a screenshot. Once I figured out which image tiles mapped to which in-game cell it was relatively easy to put a map together by plugging them into the Mapbox map as a raster tile source.

The heatmap overlaid on the map is created using a Mapbox layer that fills a cell with a color on a gradient from green to red depending on how many edits that cell has across the whole database of mods.

The sidebar on the site is created using React and Redux and uses the next/router to keep track of which page the user is on with URL parameters.

The mod search is implemented using MiniSearch that asynchronously loads the giant search indices for each game containing every mod name and id.

One of the newest features of the site allows users to drill down to a particular plugin within a file of a mod and “Add” it to their list. All of the added plugins will be listed in the sidebar and the cells they edit displayed in purple outlines and conflicts between them displayed in red outlines.

Loading plugins client-side with WebAssembly

A feature that many users requested after the initial release was being able to load a list of the mods currently installed on their game and see which ones of that set conflict with each other⁹. Implementing this feature was one of the most interesting parts of the project. Choosing to use Rust made made it possible, since everything I was running server-side to extract the plugin data could also be done client-side in the browser with the same Rust code compiled to WebAssembly.

I used wasm-pack to create skyrim-cell-dump-wasm which exported the parse_plugin function from my skyrim-cell-dump Rust library compiled to WebAssembly. It also exports a hash_plugin function that creates a unique hash for a plugin file’s slice of bytes using seahash so the site can link plugins a user has downloaded on their hard-drive to plugins that have been downloaded by modmapper and saved in the database.

Dragging-and-dropping the Skyrim Data folder on to the webpage or selecting the folder in the “Open Skyrim Data directory” dialog kicks off a process that starts parsing all of the plugin files in that directory in parallel using Web Workers.

I developed my own WorkerPool that manages creating a pool of available workers and assigns them to plugins to process. The pool size is the number of cores on the user’s device so that the site can process as many plugins in parallel as possible. After a plugin finishes processing a plugin and sends the output to the redux store, it gets added back to the pool and is then assigned a new plugin to process if there are any¹⁰.

Once all plugins have been loaded, the map updates by displaying all of the cells edited in a purple box and any cells that are edited by more than one plugin in a red box.

Users can also drag-and-drop or paste their plugins.txt file, which is the file that the game uses to define the load order of plugins and which plugins are enabled or disabled. Adding the plugins.txt sorts the list of loaded plugins in the sidebar in load order and enables or disables plugins as defined in the plugins.txt.

Selecting a cell in the map will display all of the loaded cells that edit that cell in the sidebar.

The ability to load plugins straight from a user’s hard-drive allows users to map mods that haven’t even been uploaded to Nexus Mods.

Vortex integration

The initial mod I released on the Skyrim Special Edition page of Nexus Mods was taken down by the site admins since it didn’t contain an actual mod and they didn’t agree that it qualified as a “Utility”.

Determined to have an actual mod page for Modmapper on Nexus Mods, I decided to make a Vortex integration for modmapper. Vortex is a mod manager made by the developers of Nexus Mods and they allow creating extensions to the tool and have their own mod section for Vortex extensions.

With the help of Pickysaurus, one of the community managers for Nexus Mods, I created a Vortex integration for Modmapper. It adds a context menu option on mods to view the mod in Modmapper with all of the cells it edits selected in purple. It also adds a button next to every plugin file to view just that plugin in Modmapper (assuming it has been processed by Modmapper).

To enable the latter part, I had to include skyrim-cell-dump-wasm in the extension so that I could hash the plugin contents with seahash to get the same hash that Modmapper would have generated. It only does this hashing when you click the “See on Modmapper” button to save from excessive CPU usage when viewing the plugin list.

After releasing the Vortex plugin, Pickysaurus published a news article about modmapper to the Skyrim Special Edition site which also got a lot of nice comments ❤️.

Finishing the collection by adding all Skyrim Classic mods

Skyrim is very silly in that it has many editions. But there was only one that split the modding universe into two: Skyrim Special Edition (SE). It was released in October 2016 with a revamped game engine that brought some sorely needed graphical upgrades. However, it also contained changes to how mods worked, requiring all mod authors to convert their mods to SE. This created big chasm in the library of mods, and Nexus Mods had to make a separate section for SE-only mods.

When I started downloading mods in 2021, I started only with Skyrim SE mods, which, at the time of writing, totals at over 55,000 mods on Nexus Mods.

After releasing with just SE mods, many users requested that all of the classic pre-SE Skyrim mods be added as well. This month, I finally finished downloading all Skyrim Classic mods, which, at the time of writing, totals at over 68,000 mods on Nexus Mods. That brings the total downloaded and processed mods for Modmapper at over 113,000 mods⁴!

The future

A lot of users had great feedback and suggestions on what to add to the site. I could only implement so many of them, though. The rest I’ve been keeping track of on this Trello board.

Some of the headline items on it are:

Add Solstheim map

Since map tiles images are available for that worldspace and because I have already recorded edits to the worldspace in my database, it shouldn’t be too terribly difficult.
Add Mod Organizer 2 plugin

Lots of people requested this since it’s a very popular mod manager compared to Vortex. MO2 supports python extensions so I created skyrim-cell-dump-py to export the Rust plugin processing code to a Python library. I got a bit stuck on actually creating the plugin though, so it might be a while until I get to that.
Find a way to display interior cell edits on the map

The map is currently missing edits to interior cells. Since almost all interior cells in Skyrim have a link to the exterior world through a door teleporter, it should be possible to map an interior cell edit to an exterior cell on the map based on which cell the door leads out to.

That will require digging much more into the plugin files for more data, and city worldspaces will complicate things further. Then there’s the question of interiors with multiple doors to different exterior cells, or interior cells nested recursively deep within many other interior cells.
Create a standalone Electron app that can run outside the browser

I think this would solve a lot of the issues I ran into while developing the website. Since Electron has a Node.js process running on the user’s computer outside the sandboxed browser process, it gives me much more flexibility. It could do things like automatically load a user’s plugin files. Or just load plugins at all wihtout having to deal with the annoying dialog that lies to the user saying they’re about to upload their entire Data folder hundreds of gigabytes full of files to a server (I really wish the HTMLInputElement.webkitdirectory API would use the same underlying code as the HTML Drag and Drop API which is a lot better).
Improving the search

The mod search feature struggles the most with the static generated nature of the site. I found it very hard to pack all of the necessary info for the search index for all 100k+ mods (index for both SE and LE is around 6 MB). Asynchronously loading the indices with MiniSearch keeps it from freezing up the browser, but it does take a very long time to fully load. I can’t help think that there’s a better way to shard the indices somehow and only fetch what I need based on what the user is typing into the search.

To be clear, a lot of the Todos on the board are pipe-dreams. I may never get to them. This project is sustained purely by my motivation and self-interests and if something is too much of a pain to get working I’ll just drop it.

There will also be future Elder Scrolls games, and future Bethesda games based on roughly the same game engine. It would be neat to create similar database for those games as the modding community develops in realtime.

Overall, I’m glad I made something of use to the modding community. I hope to keep the site running for as long as people are modding Skyrim (until the heat-death of the universe, probably).

Footnotes

Unfortunately, I basically lost interest on the mod after working on Modmapper. I might still write a blog post about it eventually since I did a lot of interesting hacking on the Skyrim game engine to try to add some asynchronous multiplayer aspects. Project is here if anyone is curious in the meantime. ↩
I sadly only have screenshots for some of the comments on that mod since it was eventually taken down by the Nexus Mod admins. See explanation about that in the Vortex integration section. ↩
As recorded by Cloudflare’s server side analytics, which may record a fair amount of bot traffic. I suspect this is the most accurate number I can get since most of my users probably use an ad blocker that blocks client-side analytics. ↩
Every mod on Nexus Mods except for adult mods since the site restricts viewing adult mods to only logged-in users and I wasn’t able to get my scraping bot to log in as a user. ↩ ↩²
Apparently my mass-downloading did not go unnoticed by the Nexus Mod admins. I think it’s technically against their terms of service to automatically download mods, but I somehow got on their good side and was spared the ban-hammer. I don’t recommend anyone else run modmapper themselves on the entire site unless you talk to the admins beforehand and get the okay from them. ↩
If you would like access to this dataset of plugins to do some data-mining please reach out to me at [email protected]">[email protected] (Note: only contains plugins files, no models, textures, audio, etc.). I don’t plan on releasing it publicly since that would surely go against many mod authors’ wishes/licenses. ↩
I’m not sure I want to recommend anyone else use Cloudflare after the whole Kiwi Farms debacle. I now regret having invested so much of the infrastructure in them. However, I’m only using their free-tier, so at least I am a net-negative for their business? I would recommend others look into Netlify or fastly for similar offerings to Cloudflare pages/CDN. ↩
I also tried to add a raster Terrain-DEM source for rendering the terrain in 3D. I got fairly far generating my own DEM RGB tiles from an upscaled greyscale heightmap constructed from the LAND records in Skyrim.esm (view it here). But, it came out all wrong: giant cliffs in the middle of the map and tiny spiky lumps with big jumps in elevation at cell boundaries. Seemed like too much work to get right than it was worth it. ↩
This was the announcement I posted to /r/skyrimmods for this feature ↩
At first, I noticed a strange issue with re-using the same worker on different plugins multiple times. After a while (~30 reuses per worker), the processing would slow to a crawl and eventually strange things started happening (I was listening to music in my browser and it started to pop and crack). It seemed like the speed of processing increased exponentially to the number of times the worker was reused. So, to avoid this, I had to make the worker pool terminate and recreate workers after every plugin processed. This ended up not being as slow as it sounds and worked fine. However, I recently discovered that wee_alloc, the most suggested allocator to use with rust in wasm, has a memory leak and is mostly unmaintained now. I switched to the default allocator and I didn’t run into the exponentially slow re-use problem. For some reason, the first run on a fresh tab is always much faster than the second run, but subsequent runs are still fairly stable in processing time. ↩

Generating icosahedrons and hexspheres in Rust

Sat, 01 Feb 2020 00:00:00 +0000

I’ve been trying to learn Rust lately, the hot new systems programming language. One of the projects I wanted to tackle with the speed of Rust was generating 3D polyhedron shapes. Specifically, I wanted to implement something like the Three.js IcosahedronGeometry in Rust. If you try to generate icosahedrons in Three.js over any detail level over 5 the whole browser will slow to a crawl. I think we can do better in Rust!

Furthermore, I wanted to generate a hexsphere: a sphere composed of hexagon faces and 12 pentagon faces, otherwise known as a truncated icosahedron or the Goldberg polyhedron. The shape would be ideal for a game since (almost) every tile would have the same area and six sides to defend or attack from. There’s a few Javascript projects for generating hexspheres. Most of them generate the shape by starting with a subdivided icosahedron and then truncating the sides into hexagons. Though, there exist other methods for generating the hexsphere shape.

Play around with all of these shapes in your browser at: https://www.hallada.net/planet/.

So, how would we go about generating a hexsphere from scratch?

The Icosahedron Seed

To start our sculpture, we need our ball of clay. The most basic shape that we start with can be defined by its 20 triangle faces and 12 vertices: the regular icosahedron. If you’ve ever played Dungeons and Dragons, this is the 20-sided die.

To define this basic shape in Rust, we first need to define a few structs. The most basic unit we need is a 3D vector which describes a single point in 3D space with a X, Y, and Z float values. I could have defined this myself, but to avoid having to implement a bunch of vector operations (like add, subtract, multiply, etc.) I chose to import Vector3 from the cgmath crate.

The next struct we need is Triangle. This will define a face between three vertices:

#[derive(Debug)]
pub struct Triangle {
    pub a: usize,
    pub b: usize,
    pub c: usize,
}

impl Triangle {
    fn new(a: usize, b: usize, c: usize) -> Triangle {
        Triangle { a, b, c }
    }
}

We use usize for the three points of the triangle because they are indices into a Vec of Vector3s.

To keep these all together, I’ll define a Polyhedron struct:

#[derive(Debug)]
pub struct Polyhedron {
    pub positions: Vec<Vector3>,
    pub cells: Vec<Triangle>,
}

With this, we can define the regular icosahedron:

impl Polyhedron {
    pub fn regular_isocahedron() -> Polyhedron {
        let t = (1.0 + (5.0 as f32).sqrt()) / 2.0;
        Polyhedron {
            positions: vec![
                Vector3::new(-1.0, t, 0.0),
                Vector3::new(1.0, t, 0.0),
                Vector3::new(-1.0, -t, 0.0),
                Vector3::new(1.0, -t, 0.0),
                Vector3::new(0.0, -1.0, t),
                Vector3::new(0.0, 1.0, t),
                Vector3::new(0.0, -1.0, -t),
                Vector3::new(0.0, 1.0, -t),
                Vector3::new(t, 0.0, -1.0),
                Vector3::new(t, 0.0, 1.0),
                Vector3::new(-t, 0.0, -1.0),
                Vector3::new(-t, 0.0, 1.0),
            ],
            cells: vec![
                Triangle::new(0, 11, 5),
                Triangle::new(0, 5, 1),
                Triangle::new(0, 1, 7),
                Triangle::new(0, 7, 10),
                Triangle::new(0, 10, 11),
                Triangle::new(1, 5, 9),
                Triangle::new(5, 11, 4),
                Triangle::new(11, 10, 2),
                Triangle::new(10, 7, 6),
                Triangle::new(7, 1, 8),
                Triangle::new(3, 9, 4),
                Triangle::new(3, 4, 2),
                Triangle::new(3, 2, 6),
                Triangle::new(3, 6, 8),
                Triangle::new(3, 8, 9),
                Triangle::new(4, 9, 5),
                Triangle::new(2, 4, 11),
                Triangle::new(6, 2, 10),
                Triangle::new(8, 6, 7),
                Triangle::new(9, 8, 1),
            ],
        }
    }
}

JSON Serialization

To prove this works, we need to be able to output our shape to some format that will be able to be rendered. Coming from a JS background, I’m only familiar with rendering shapes with WebGL. So, I need to be able to serialize the shape to JSON so I can load it in JS.

There’s an amazing library in Rust called serde that will make this very straightforward. We just need to import it and impl Serialize for all of our structs.

The JSON structure we want will look like this. This is what Three.js expects when initializing BufferGeometry.

{
    "positions": [
        [
          -0.8506508,
          0,
          0.5257311
        ],
        ...
    ],
    "cells": [
        [
            0,
            1,
            2,
        ],
        ...
    ],
}

For the "cells" array, we’ll need to serialize Triangle into an array of 3 integer arrays:

impl Serialize for Triangle {
    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
    where
        S: Serializer,
    {
        let vec_indices = vec![self.a, self.b, self.c];
        let mut seq = serializer.serialize_seq(Some(vec_indices.len()))?;
        for index in vec_indices {
            seq.serialize_element(&index)?;
        }
        seq.end()
    }
}

I had some trouble serializing the cgmath::Vector3 to an array, so I made my own type that wrapped Vector3 that could be serialized to an array of 3 floats.

#[derive(Debug)]
pub struct ArraySerializedVector(pub Vector3<f32>);

impl Serialize for ArraySerializedVector {
    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
    where
        S: Serializer,
    {
        let values = vec![self.0.x, self.0.y, self.0.z];
        let mut seq = serializer.serialize_seq(Some(values.len()))?;
        for value in values {
            seq.serialize_element(&value)?;
        }
        seq.end()
    }
}

And now Polyhedron needs to use this new type and implement Serialize for the whole shape to get serialized:

#[derive(Serialize, Debug)]
pub struct Polyhedron {
    pub positions: Vec<ArraySerializedVector>,
    pub cells: Vec<Triangle>,
}

The actual serialization is done with:

fn write_to_json_file(polyhedron: Polyhedron, path: &Path) {
    let mut json_file = File::create(path).expect("Can't create file");
    let json = serde_json::to_string(&polyhedron).expect("Problem serializing");
    json_file
        .write_all(json.as_bytes())
        .expect("Can't write to file");
}

On the JS side, the .json file can be read and simply fed into either Three.js or regl to be rendered in WebGL (more on that later).

Subdivided Icosahedron

Now, we need to take our regular icosahedron and subdivide its faces N number of times to generate an icosahedron with a detail level of N.

I pretty much copied must of the subdividing code from Three.js directly into Rust.

I won’t bore you with the details here, you can find the function here.

Truncated Icosahedron

Now we get to the meat of this project. Transforming an icosahedron into a hexsphere by truncating the points of the icosahedron into hexagon and pentagon faces.

You can imagine this operation as literally cutting off the points of the subdivided icosahedron at exactly the midpoint between the point and it’s six or five neighboring points.

(image source)

In this image you can see the regular icosahedron (0 subdivisions) in wireframe with a yellow shape underneath which is the result of all 12 points truncated to 12 pentagon faces, in other words: the regular dodecahedron.

You can see that the points of the new pentagon faces will be the exact center of the original triangular faces. It should now make sense why truncating a shape with 20 faces of 3 edges each results in a shape with 12 faces of 5 edges each. Each pair multiplied still equals 60.

Algorithm

There are many different algorithms you could use to generate the truncated shape, but this is roughly what I came up with:

Store a map of every icosahedron vertex to faces composed from that vertex (vert_to_faces).
Calculate and cache the centroid of every triangle in the icosahedron (triangle_centroids).
For every vertex in the original icosahedron:
Find the center point between all of centroids of all of the faces for that vertex (center_point). This is essentially the original icosahedron point but lowered towards the center of the polygon since it will eventually be the center of a new flat hexagon face.
For every triangle face composed from the original vertex:
Sort the vertices of the triangle face so there is a vertex A in the center of the fan like in the image, and two other vertices B and C at the edges of the hexagon.
Find the centroid of the selected face. This will be one of the five or six points of the new pentagon or hexagon (in brown in diagram below: triangleCentroid).
Find the mid point between AB and AC (points midAB and midAC in diagram).
With these mid points and the face centroid, we now have two new triangles (in orange below) that form one-fifth or one-sixth of the final pentagon or hexagon face. Add the points of the triangle to the positions array. Add the two new triangles composed from those vertices as indexes into the positions array to the cells array. We need to compose the pentagon or hexagon out of triangles because in graphics everything is a triangle, and this is the simplest way to tile either shape with triangles:
Go to step 5 until all faces of the icosahedron vertex have been visited.
Save indices to all new triangles in the cells array, which now form a complete pentagon or hexagon face, to the faces array.
Go to step 3 until all vertices in the icosahedron have been visited. The truncated icosahedron is now complete.

Code

The truncate function calls out to a bunch of other functions, so here’s a link to the function within the context of the whole file.

Calculating Normals

It took me a surprisingly long time to figure out how to compute normals for the truncated icosahedron. I tried just using an out-of-the-box solution like angle-normals which could supposedly calculate the normal vectors for you, but they came out all wrong.

So, I tried doing it myself. Most tutorials on computing normal vectors for a mesh assume that it is tiled in a particular way. But, my algorithm spins around icosahedron points in all different directions, and so the triangle points are not uniformly in clockwise or counter-clockwise order.

I could have sorted these points into the correct order, but I found it easier to instead just detect when the normal was pointing the wrong way and just invert it.

pub fn compute_triangle_normals(&mut self) {
    let origin = Vector3::new(0.0, 0.0, 0.0);
    for i in 0..self.cells.len() {
        let vertex_a = &self.positions[self.cells[i].a].0;
        let vertex_b = &self.positions[self.cells[i].b].0;
        let vertex_c = &self.positions[self.cells[i].c].0;

        let e1 = vertex_a - vertex_b;
        let e2 = vertex_c - vertex_b;
        let mut no = e1.cross(e2);

        // detect and correct inverted normal
        let dist = vertex_b - origin;
        if no.dot(dist) < 0.0 {
            no *= -1.0;
        }

        let normal_a = self.normals[self.cells[i].a].0 + no;
        let normal_b = self.normals[self.cells[i].b].0 + no;
        let normal_c = self.normals[self.cells[i].c].0 + no;

        self.normals[self.cells[i].a] = ArraySerializedVector(normal_a);
        self.normals[self.cells[i].b] = ArraySerializedVector(normal_b);
        self.normals[self.cells[i].c] = ArraySerializedVector(normal_c);
    }

    for normal in self.normals.iter_mut() {
        *normal = ArraySerializedVector(normal.0.normalize());
    }
}

Assigning Random Face Colors

Finally, all that’s left to generate is the face colors. The only way I could figure out how to individually color a shape’s faces in WebGL was to pass a color per vertex. The issue with this is that each vertex of the generated shapes could be shared between many different faces.

How can we solve this? At the cost of memory, we can just duplicate a vertex every time it’s used by a different triangle. That way no vertex is shared.

This can be done after a shape has been generated with shared vertices.

pub fn unique_vertices(&mut self, other: Polyhedron) {
    for triangle in other.cells {
        let vertex_a = other.positions[triangle.a].0;
        let vertex_b = other.positions[triangle.b].0;
        let vertex_c = other.positions[triangle.c].0;
        let normal_a = other.normals[triangle.a].0;
        let normal_b = other.normals[triangle.b].0;
        let normal_c = other.normals[triangle.c].0;

        self.positions.push(ArraySerializedVector(vertex_a));
        self.positions.push(ArraySerializedVector(vertex_b));
        self.positions.push(ArraySerializedVector(vertex_c));
        self.normals.push(ArraySerializedVector(normal_a));
        self.normals.push(ArraySerializedVector(normal_b));
        self.normals.push(ArraySerializedVector(normal_c));
        self.colors
            .push(ArraySerializedVector(Vector3::new(1.0, 1.0, 1.0)));
        self.colors
            .push(ArraySerializedVector(Vector3::new(1.0, 1.0, 1.0)));
        self.colors
            .push(ArraySerializedVector(Vector3::new(1.0, 1.0, 1.0)));
        let added_index = self.positions.len() - 1;
        self.cells
            .push(Triangle::new(added_index - 2, added_index - 1, added_index));
    }
    self.faces = other.faces;
}

With unique vertices, we can now generate a random color per face with the rand crate.

pub fn assign_random_face_colors(&mut self) {
    let mut rng = rand::thread_rng();
    for i in 0..self.faces.len() {
        let face_color = Vector3::new(rng.gen(), rng.gen(), rng.gen());

        for c in 0..self.faces[i].len() {
            let face_cell = &self.cells[self.faces[i][c]];

            self.colors[face_cell.a] = ArraySerializedVector(face_color);
            self.colors[face_cell.b] = ArraySerializedVector(face_color);
            self.colors[face_cell.c] = ArraySerializedVector(face_color);
        }
    }
}

Binary Serialization

Now that we have to duplicate vertices for individual face colors, the size of our JSON outputs are getting quite big:

File	Size
icosahedron_r1_d6.json	28 MB
icosahedron_r1_d7.json	113 MB
hexsphere_r1_d5.json	42 MB
hexsphere_r1_d6.json	169 MB

Since all of our data is just floating point numbers, we could reduce the size of the output considerably by using a binary format instead.

I used the byteorder crate to write out all of the Vecs in my Polyhedron struct to a binary file in little-endian order.

The binary format is laid out as:

1 32 bit unsigned integer specifying the number of vertices (V)
1 32 bit unsigned integer specifying the number of triangles (T)
V * 3 number of 32 bit floats for every vertex’s x, y, and z coordinate
V * 3 number of 32 bit floats for the normals of every vertex
V * 3 number of 32 bit floats for the color of every vertex
T * 3 number of 32 bit unsigned integers for the 3 indices into the vertex array that make every triangle

The write_to_binary_file function which does all that is here.

That’s a lot better:

File	Size
icosahedron_r1_d6.bin	9.8 MB
icosahedron_r1_d7.bin	11 MB
hexsphere_r1_d5.bin	14 MB
hexsphere_r1_d6.bin	58 MB

On the JavaScript side, the binary files can be read into Float32Arrays like this:

fetch(binaryFile)
  .then(response => response.arrayBuffer())
  .then(buffer => {
    let reader = new DataView(buffer);
    let numVertices = reader.getUint32(0, true);
    let numCells = reader.getUint32(4, true);
    let shape = {
      positions: new Float32Array(buffer, 8, numVertices * 3),
      normals: new Float32Array(buffer, numVertices * 12 + 8, numVertices * 3),
      colors: new Float32Array(buffer, numVertices * 24 + 8, numVertices * 3),
      cells: new Uint32Array(buffer, numVertices * 36 + 8, numCells * 3),
    })

Rendering in WebGL with Regl

I was initially rendering the shapes with Three.js but switched to regl because it seemed like a more direct abstraction over WebGL. It makes setting up a WebGL renderer incredibly easy compared to all of the dozens cryptic function calls you’d have to otherwise use.

This is pretty much all of the rendering code using regl in my 3D hexsphere and icosahedron viewer project.

const drawShape = hexsphere => regl({
  vert: `
  precision mediump float;
  uniform mat4 projection, view;
  attribute vec3 position, normal, color;
  varying vec3 fragNormal, fragPosition, fragColor;
  void main() {
    fragNormal = normal;
    fragPosition = position;
    fragColor = color;
    gl_Position = projection * view * vec4(position, 1.0);
  }`,

  frag: `
  precision mediump float;
  struct Light {
    vec3 color;
    vec3 position;
  };
  uniform Light lights[1];
  varying vec3 fragNormal, fragPosition, fragColor;
  void main() {
    vec3 normal = normalize(fragNormal);
    vec3 light = vec3(0.1, 0.1, 0.1);
    for (int i = 0; i < 1; i++) {
      vec3 lightDir = normalize(lights[i].position - fragPosition);
      float diffuse = max(0.0, dot(lightDir, normal));
      light += diffuse * lights[i].color;
    }
    gl_FragColor = vec4(fragColor * light, 1.0);
  }`,

  attributes: {
    position: hexsphere.positions,
    normal: hexsphere.normals,
    color: hexsphere.colors,
  },
  elements: hexsphere.cells,
  uniforms: {
    "lights[0].color": [1, 1, 1],
    "lights[0].position": ({ tick }) => {
      const t = 0.008 * tick
      return [
        1000 * Math.cos(t),
        1000 * Math.sin(t),
        1000 * Math.sin(t)
      ]
    },
  },
})

I also imported regl-camera which handled all of the complex viewport code for me.

It was fairly easy to get a simple renderer working quickly in regl, but I couldn’t find many examples of more complex projects using regl. Unfortunately, the project looks a bit unmaintained these days as well. If I’m going to continue with rendering in WebGL, I think I will try out Babylon.js instead.

Running in WebAssembly

Since rust can be compiled down to wasm and then run in the browser, I briefly tried getting the project to run completely in the browser.

The wasm-pack tool made it pretty easy to get started. My main struggle was figuring out an efficient way to get the megabytes of generated shape data into the JavaScript context so it could be rendered in WebGL.

The best I could come up with was to export all of my structs into flat Vec<f32>s and then create Float32Arrays from the JS side that are views into wasm’s memory.

To export:

pub fn fill_exports(&mut self) {
    for position in &self.positions {
        self.export_positions.push(position.0.x);
        self.export_positions.push(position.0.y);
        self.export_positions.push(position.0.z);
    }
    for normal in &self.normals {
        self.export_normals.push(normal.0.x);
        self.export_normals.push(normal.0.y);
        self.export_normals.push(normal.0.z);
    }
    for color in &self.colors {
        self.export_colors.push(color.0.x);
        self.export_colors.push(color.0.y);
        self.export_colors.push(color.0.z);
    }
    for cell in &self.cells {
        self.export_cells.push(cell.a as u32);
        self.export_cells.push(cell.b as u32);
        self.export_cells.push(cell.c as u32);
    }
}

And then the wasm lib.rs:

use byteorder::{LittleEndian, WriteBytesExt};
use js_sys::{Array, Float32Array, Uint32Array};
use wasm_bindgen::prelude::*;
use web_sys::console;

mod icosahedron;

#[cfg(feature = "wee_alloc")]
#[global_allocator]
static ALLOC: wee_alloc::WeeAlloc = wee_alloc::WeeAlloc::INIT;

#[wasm_bindgen(start)]
pub fn main_js() -> Result<(), JsValue> {
    #[cfg(debug_assertions)]
    console_error_panic_hook::set_once();

    Ok(())
}

#[wasm_bindgen]
pub struct Hexsphere {
    positions: Float32Array,
    normals: Float32Array,
    colors: Float32Array,
    cells: Uint32Array,
}

#[wasm_bindgen]
pub fn shape_data() -> Result<Array, JsValue> {
    let radius = 1.0;
    let detail = 7;
    let mut hexsphere = icosahedron::Polyhedron::new_truncated_isocahedron(radius, detail);
    hexsphere.compute_triangle_normals();
    let mut unique_hexsphere = icosahedron::Polyhedron::new();
    unique_hexsphere.unique_vertices(hexsphere);
    unique_hexsphere.assign_random_face_colors();
    unique_hexsphere.fill_exports();

    let positions = unsafe { Float32Array::view(&unique_hexsphere.export_positions) };
    let normals = unsafe { Float32Array::view(&unique_hexsphere.export_normals) };
    let colors = unsafe { Float32Array::view(&unique_hexsphere.export_colors) };
    let cells = unsafe { Uint32Array::view(&unique_hexsphere.export_cells) };

    Ok(Array::of4(&positions, &normals, &colors, &cells))
}

With wasm-pack, I could import the wasm package, run the shape_data() function, and then read the contents as any other normal JS array.

let rust = import("../pkg/index.js")
rust.then(module => {
    const shapeData = module.shape_data()
    const shape = {
        positions: shapeData[0],
        normals: shapeData[1],
        colors: shapeData[2],
        cells: shapeData[3],
    }
    ...
})

I could side-step the issue of transferring data from Rust to JavaScript entirely by programming literally everything in WebAssembly. But the bindings from rust wasm to the WebGL API are still way too complicated compared to just using regl. Plus, I’d have to implement my own camera from scratch.

The Stats

So how much faster is Rust than JavaScript in generating icosahedrons and hexspheres?

Here’s how long it took with generating shapes in JS with Three.js in Firefox versus in native Rust with a i5-2500K 3.3 GHz CPU.

Shape	JS generate time	Rust generate time
Icosahedron detail 6	768 ms	28.23 ms
Icosahedron detail 7	4.25 s	128.81 ms
Hexsphere detail 6	11.37 s	403.10 ms
Hexsphere detail 7	25.49 s	1.85 s

So much faster!

Todo

Add a process that alters the shape post-generation. Part of the reason why I decided to fan the hexagon faces with so many triangles is that it also allows me to control the height of the faces better. This could eventually allow me to create mountain ranges and river valleys on a hexsphere planet. Stretching and pulling the edges of the polygon faces in random directions could add variation and make for a more organic looking hexsphere.
Conversely, it would be nice to be able to run a process post-generation that could reduce the number of triangles by tiling the hexagons more efficiently when face elevation isn’t needed.
Add parameters to the generation that allows generating sections of the hexsphere / icosahedron. This will be essential for rendering very detailed polyhedrons since at a certain detail level it becomes impossible to render the entire shape at once.

In WebGL, figure out what part of the shape is in the current viewport and pass these parameters to the generation.
Render the shapes in a native Rust graphics library instead of WebGL. I’m curious how much slower WebGL is making things.
Parallelize the generation. Right now the generation is very CPU bound and each subdivide/truncate iteration is mostly independent from each other, so I think I could get some decent speed-up by allowing the process to run on multiple cores. Perhaps the rayon crate could make this pretty straightforward.
Find some way to avoid unique vertices. The size of the shape is much bigger because of this. There might be a way to keep shared vertices while also having a separate color per face by using texture mapping.
In the renderer, implement face selection (point and click face and show an outline around selected face).
In the renderer, implement fly-to-face zooming: given a face, fly the camera around the sphere in an orbit and then zoom in on the face.

Studio-Frontend: Developing Frontend Separate from edX Platform

Thu, 26 Apr 2018 00:00:00 +0000

This is a blog post that I originally wrote for the edX engineering blog.

At the core of edX is the edx-platform, a monolithic Django code-base 2.7 times the size of Django itself.

-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 ActionScript            1          118           74           23           21
 Autoconf               10          425          237          163           25
 CSS                    55        17106        14636         1104         1366
 HTML                  668        72567        36865        30306         5396
 JavaScript           1500       463147       352306        55882        54959
 JSON                   91        14583        14583            0            0
 JSX                    33         2595         2209           62          324
 LESS                    1          949          606          232          111
 Makefile                1           65           49            8            8
 Markdown               23          287          287            0            0
 Mustache                1            1            1            0            0
 Python               3277       559255       442756        29254        87245
 ReStructuredText       48         4252         4252            0            0
 Sass                  424        75559        55569         4555        15435
 Shell                  15          929          505          292          132
 SQL                     4         6283         5081         1186           16
 Plain Text            148         3521         3521            0            0
 TypeScript             20        88506        76800        11381          325
 XML                   364         5283         4757          231          295
 YAML                   36         1630         1361          119          150
-------------------------------------------------------------------------------
 Total                6720      1317061      1016455       134798       165808
-------------------------------------------------------------------------------

35% of the edx-platform is JavaScript. While it has served edX well since its inception in 2012, reaching over 11 million learners in thousands of courses on edX.org and many more millions on all of the Open edX instances across the world, it is starting to show its age. Most of it comes in the form of Backbone.js apps loaded by RequireJS in Django Mako templates, with jQuery peppered throughout.

Many valiant efforts are underway to modernize the frontend of edx-platform including replacing RequireJS with Webpack, Backbone.js with React, and ES5 JavaScript and CoffeeScript with ES6 JavaScript. Many of these efforts were covered in detail at the last Open edX conference and in Open edX Proposal 11: Front End Technology Standards. However, the size and complexity of the edx-platform means that these kind of efforts are hard to prioritize, and, in the meantime, frontend developers are forced to wait over 10 minutes for our home-grown asset pipeline to build before they can view changes.

There have also been efforts to incrementally modularize and extract parts of the edx-platform into separate python packages that could be installed as Django apps, or even as separately deployed microservices. This allows developers to work independently from the rest of the organization inside of a repository that they own, manage, and is small enough that they could feasibly understand it entirely.

When my team was tasked with improving the user experience of pages in Studio, the tool that course authors use to create course content, we opted to take a similar architectural approach with the frontend and create a new repository where we could develop new pages in isolation and then integrate them back into the edx-platform as a plugin. We named this new independent repository studio-frontend. With this approach, our team owns the entire studio-frontend code-base and can make the best architectural changes required for its features without having to consult with and contend with all of the other teams at edX that contribute to the edx-platform. Developers of studio-frontend can also avoid the platform’s slow asset pipeline by doing all development within the studio-frontend repository and then later integrating the changes into platform.

React and Paragon

When edX recently started to conform our platform to the Web Content Accessibility Guidelines 2.0 AA (WCAG 2.0 AA), we faced many challenges in retrofitting our existing frontend code to be accessible. Rebuilding Studio pages from scratch in studio-frontend allows us to not only follow the latest industry standards for building robust and performant frontend applications, but to also build with accessibility in mind from the beginning.

The Javascript community has made great strides recently to address accessibility issues in modern web apps. However, we had trouble finding an open-source React component library that fully conformed to WCAG 2.0 AA and met all of edX’s needs, so we decided to build our own: Paragon.

Paragon is a library of building-block components like buttons, inputs, icons, and tables which were built from scratch in React to be accessible. The components are styled using the Open edX theme of Bootstrap v4 (edX’s decision to adopt Bootstrap is covered in OEP-16). Users of Paragon may also choose to use the themeable unstyled target and provide their own Bootstrap theme.

Studio-frontend composes together Paragon components into higher-level components like an accessibility form or a table for course assets with searching, filtering, sorting, pagination, and upload. While we developed these components in studio-frontend, we were able to improve the base Paragon components. Other teams at edX using the same components were able to receive the same improvements with a single package update.

Integration with Studio

We were able to follow the typical best practices for developing a React/Redux application inside studio-frontend, but at the end of the day, we still had to somehow get our components inside of existing Studio pages and this is where most of the challenges arose.

Webpack

The aforementioned move from RequireJS to Webpack in the edx-platform made it possible for us to build our studio-frontend components from source with Webpack within edx-platform. However, this approach tied us to the edx-platform’s slow asset pipeline. If we wanted rapid development, we had to duplicate the necessary Webpack config between both studio-frontend and edx-platform.

Instead, studio-frontend handles building the development and production Webpack builds itself. In development mode, the incremental rebuild that happens automatically when a file is changed takes under a second. The production JavaScript and CSS bundles, which take about 25 seconds to build, are published with every new release to NPM. The edx-platform npm installs studio-frontend and then copies the built production files from node_modules into its Django static files directory where the rest of the asset pipeline will pick it up.

To actually use the built JavaScript and CSS, edx-platform still needs to include it in its Mako templates. We made a Mako template tag that takes a Webpack entry point name in studio-frontend and generates script tags that include the necessary files from the studio-frontend package. It also dumps all of the initial context that studio-frontend needs from the edx-platform Django app into a JSON object in a script tag on the page that studio-frontend components can access via a shared id. This is how studio-frontend components get initial data from Studio, like which course it’s embedded in.

For performance, modules that are shared across all studio-frontend components are extracted into common.min.js and common.min.css files that are included on every Studio template that has a studio-frontend component. User’s browsers should cache these files so that they do not have to re-download libraries like React and Redux every time they visit a new page that contains a studio-frontend component.

CSS Isolation

Since the move to Bootstrap had not yet reached the Studio part of the edx-platform, most of the styling clashed with the Bootstrap CSS that studio-frontend components introduced. And, the Bootstrap styles were also leaking outside of the studio-frontend embedded component div and affecting the rest of the Studio page around it.

We were able to prevent styles leaking outside of the studio-frontend component by scoping all CSS to only the div that wraps the component. Thanks to the Webpack postcss-loader and the postcss-prepend-selector we were able to automatically scope all of our CSS selectors to that div in our build process.

Preventing the Studio styles from affecting our studio-frontend component was a much harder problem because it means avoiding the inherently cascading nature of CSS. A common solution to this issue is to place the 3rd-party component inside of an iframe element, which essentially creates a completely separate sub-page where both CSS and JavaScript are isolated from the containing page. Because iframes introduce many other performance and styling issues, we wanted to find a different solution to isolating CSS.

The CSS style all: initial allows resetting all properties on an element to their initial values as defined in the CSS spec. Placing this style under a wildcard selector in studio-frontend allowed us to reset all inherited props from the legacy Studio styles without having to enumerate them all by hand.

* {
  all: initial;
}

While this CSS property doesn’t have broad browser support yet, we were able to polyfill it thanks to postcss with the postcss-initial plugin.

However, this resets the styles to nothing. For example, all divs are displayed inline. To return the styles back to to some sane browser default we had to re-apply a browser default stylesheet. You can read more about this technique at default-stylesheet.

From there, Bootstrap’s reboot normalizes the browser-specific styling to a common baseline and then applies the Bootstrap styles conflict-free from the surrounding CSS cascade.

There’s a candidate recommendation in CSS for a contains property, which will “allow strong, predictable isolation of a subtree from the rest of the page”. I hope that it will provide a much more elegant solution to this problem once browsers support it.

Internationalization

Another major challenge with separating out the frontend from edx-platform was that most of our internationalization tooling was instrumented inside the edx-platform. So, in order to display text in studio-frontend components in the correct language we either had to pass already-translated strings from the edx-platform into studio-frontend, or set-up translations inside studio-frontend.

We opted for the latter because it kept the content close to the code that used it. Every display string in a component is stored in a displayMessages.jsx file and then imported and referenced by an id within the component. A periodic job extracts these strings from the project, pushes them up to our translations service Transifex, and pulls any new translations to store them in our NPM package.

Because Transifex’s KEYVALUEJSON file format does not allow for including comments in the strings for translation, Eric created a library called reactifex that will send the comments in separate API calls.

Studio includes the user’s language in the context that it sends a studio-frontend component for initialization. Using this, the component can display the message for that language if it exists. If it does not, then it will display the original message in English and wrap it in a span with lang="en" as an attribute so that screen-readers know to read it in English even if their default is some other language.

Read more about studio-frontend’s internationalization process in the documentation that Eric wrote.

Developing with Docker

To normalize the development environment across the whole studio-frontend team, development is done in a Docker container. This is a minimal Ubuntu 16.04 container with specific version of Node 8 installed and its only purpose is to run Webpack. This follows the pattern established in OEP-5: Pre-built Development Environments for running a single Docker container per process that developers can easily start without installing dependencies.

Similar to edX’s devstack there is a Makefile with commands to start and stop the docker container. The docker container then immediately runs npm run start, which runs Webpack with the webpack-dev-server. The webpack-dev-server is a node server that serves assets built by Webpack. Studio-frontend’s Webpack config makes this server available to the developer’s host machine at http://localhost:18011.

With hot-reload enabled, developers can now visit that URL in their browser, edit source files in studio-frontend, and then see changes reflected instantly in their browser once Webpack finishes its incremental rebuild.

However, many studio-frontend components need to be able to talk to the edx-platform Studio backend Django server. Using docker’s network connect feature the studio-frontend container can join the developer’s existing docker devstack network so that the studio-frontend container can make requests to the docker devstack Studio container at http://edx.devstack.studio:18010/ and Studio can access studio-frontend at http://dahlia.studio-fronend:18011/.

The webpack-dev-server can now proxy all requests to Studio API endpoints (like http://localhost:18011/assets) to http://edx.devstack.studio:18010/.

Developing within Docker Devstack Studio

Since studio-frontend components will be embedded inside of an existing Studio page shell, it’s often useful to develop on studio-frontend containers inside of this set-up. This can be done by setting a variable in the devstack’s cms/envs/private.py:

STUDIO_FRONTEND_CONTAINER_URL = 'http://localhost:18011'

This setting is checked in the Studio Mako templates wherever studio-frontend components are embedded. If it is set to a value other than None, then the templates will request assets from that URL instead of the Studio’s own static assets directory. When a developer loads a Studio page with an embedded studio-frontend component, their studio-frontend webpack-dev-server will be requested at that URL. Similarly to developing on studio-frontend in isolation, edits to source files will trigger a Webpack compilation and the Studio page will be hot-reloaded or reloaded to reflect the changes automatically.

Since the studio-frontend JS loaded on localhost:18010 is now requesting the webpack-dev-server on localhost:18011, an Access-Control-Allow-Origin header has to be configured on the webpack-dev-server to get around CORS violations.

Deploying to Production

Each release of studio-frontend will upload the /dist files built by Webpack in production mode to NPM. edx-platform requires a particular version of studio-frontend in its package.json. When a new release of edx-platform is made, paver update_assets will run which will copy all of the files in the node_modules/@edx/studio-frontend/dist/ to the Studio static folder. Because STUDIO_FRONTEND_CONTAINER_URL will be None in production, it will be ignored, and Studio pages will request studio-frontend assets from that static folder.

Future

Instead of “bringing the new into the old”, we’d eventually like to move to a model where we “work in the new and bring in the old if necessary”. We could host studio-frontend statically on a completely separate server which talks to Studio via a REST (or GraphQL) API. This approach would eliminate the complexity around CSS isolation and bring big performance wins for our users, but it would require us to rewrite more of Studio.

Isso Comments

Wed, 15 Nov 2017 00:00:00 +0000

I’ve been meaning to add a commenting system to this blog for a while, but I couldn’t think of a good way to do it. I implemented my own commenting system on my old Django personal site. While I enjoyed working on it at the time, it was a lot of work, especially to fight the spam. Now that my blog is hosted statically on Github’s servers, I have no way to host something dynamic like comments.

Disqus seems to be the popular solution to this problem for other people that host static blogs. The way it works is that you serve a javascript client script on the static site you own. The script will make AJAX requests to a separate server that Disqus owns to retrieve comments and post new ones.

The price you pay for using Disqus, however, is that they get to sell all of the data that you and your commenters give them. That reason, plus the fact that I wanted something more DIY, meant this blog has gone without comments for a few years.

Then I discovered Isso. Isso calls itself a lightweight alternative to Disqus. Isso allows you to install the server code on your own server so that the comment data never goes to a third party. Also, it does not require logging into some social media account just to comment. Today, I installed it on my personal AWS EC2 instance and added the Isso javascript client script on this blog. So far, my experience with it has been great and it performs exactly the way I expect.

I hit a few snags while installing it, however.

Debian Package

I don’t recommend using the Debian package anymore as it frequently goes out of date and breaks on distribution upgrades. See bottom edit.

There is a very handy Debian package that someone has made for Isso. Since my server runs Ubuntu 16.04, and Ubuntu is based off of Debian, this is a package I can install with my normal ubuntu package manager utilities. There is no PPA to install since the package is in the main Ubuntu package archive. Just run sudo apt-get install isso.

I got a bit confused after that point, though. There seems to be no documentation I could find about how to actually configure and start the server once you have installed it. This is what I did:

sudo cp /etc/default/isso /etc/isso.d/available/isso.cfg
sudo ln -s /etc/isso.d/available/isso.cfg /etc/isso.d/enabled/isso.cfg

Then you can edit /etc/isso.d/available/isso.cfg with your editor of choice to configure the Isso server for your needs. Make sure to set the host variable to the URL for your static site.

Once you’re done, you can run sudo service isso restart to reload the server with the new configuration. sudo service isso status should report Active (running).

Right now, there should be a gunicorn process running the isso server. You can check that with top or running ps aux | grep gunicorn, which should return something about “isso”.

Nginx Reverse Proxy

In order to map the URL “comments.hallada.net” to this new gunicorn server, I need an nginx reverse proxy.

To do that, I made a new server block: sudo vim /etc/nginx/sites-available/isso which I added:

server {
    listen 80;
    listen [::]:80;
    server_name comments.hallada.net;

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Script-Name /isso;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://localhost:8000;
    }
}

Then I enabled this new server block with:

sudo ln -s /etc/nginx/sites-available/isso /etc/nginx/sites-enabled/isso
sudo systemctl restart nginx

DNS Configuration

I added a new A record for “comments.hallada.net” that pointed to my server’s IP address to the DNS configuration for my domain (which I recently switched to Amazon Route 53).

After the DNS caches had time to refresh, visiting http://comments.hallada.net would hit the new isso nginx server block, which would then pass the request on to the gunicorn process.

You can verify if nginx is getting the request by looking at /var/log/nginx/access.log.

Adding the Isso Script to my Jekyll Site

I created a file called _includes/comments.html with the contents that the Isso documentation provides. Then, in my post template, I simply included that on the page where I wanted the comments to go:

<div class="card">
  <div class="row clearfix">
    <div class="column full">
      <script data-isso="https://comments.hallada.net/"
              src="https://comments.hallada.net/js/embed.min.js"></script>

      <section id="isso-thread">
        <noscript>Javascript needs to be activated to view comments.</noscript>
      </section>
    </div>
  </div>
</div>

Another thing that was not immediately obvious to me is that the value of the name variable in the Isso server configuration is the URL path that you will need to point the Isso JavaScript client to. For example, I chose name = blog, so the data-isso attribute on the script tag needed to be http://comments.hallada.net/blog/.

The Uncaught ReferenceError

You won’t need to fix this if you install Isso from PIP! See bottom edit.

There’s an issue with that Debian package that causes a JavaScript error in the console when trying to load the Isso script in the browser. I solved this by uploading the latest version of the Isso embeded.min.js file to my server, which I put at /var/www/html/isso/embeded.min.js. Then I modified the nginx server block to serve that file when the path matches /isso:

server {
    listen 80;
    listen [::]:80;
    server_name comments.hallada.net;
    root /var/www/html;

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Script-Name /isso;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://localhost:8000;
    }

    location /isso {
        try_files $uri $uri/ $uri.php?$args =404;
    }
}

Now requesting http://comments.hallada.net/isso/embeded.min.js would return the newer script without the bug.

Sending Emails Through Amazon Simple Email Service

I already set up Amazon’s SES in my last blog post. To get Isso to use SES to send notifications about new comments, create a new credential in the SES UI, and then set the user and password fields in the isso.cfg to what get’s generated for the IAM user. The SES page also has information for what host and port to use. I used security = starttls and port = 587. Make sure whatever email you use for from is a verified email in SES. Also, don’t forget to add your email as the to value.

Enabling HTTPS with Let’s Encrypt

Let’s Encrypt allows you to get SSL certificates for free! I had already installed the certbot/letsencrypt client before, so I just ran this to generate a new certificate for my new sub-domain “comments.hallada.net”:

sudo letsencrypt certonly --nginx -d comments.hallada.net

Once that successfully completed, I added a new nginx server block for the https version at /etc/nginx/sites-available/isso-https:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name comments.hallada.net;
    root /var/www/html;

    ssl_certificate /etc/letsencrypt/live/comments.hallada.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/comments.hallada.net/privkey.pem;
    ssl_trusted_certificate /etc/letsencrypt/live/comments.hallada.net/fullchain.pem;

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Script-Name /isso;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://localhost:8000;
    }

    location /isso {
        try_files $uri $uri/ $uri.php?$args =404;
    }
}

And, I changed the old http server block so that it just permanently redirects to the https version:

server {
    listen 80;
    listen [::]:80;
    server_name comments.hallada.net;
    root /var/www/html;

    location / {
        return 301 https://comments.hallada.net$request_uri;
    }
}

Then I enabled the https version:

sudo ln -s /etc/nginx/sites-available/isso-https /etc/nginx/sites-enabled/isso-https
sudo systemctl restart nginx

I checked that I didn’t get any errors visiting https://comments.hallada.net/, and then changed my Jekyll include snippet so that it pointed at the https site instead of http.

Now you can securely leave a comment if you want to yell at me for writing the wrong thing!

EDIT 5/28/2019:

I don’t recommend using the Debian package anymore since it frequently goes out of date and breaks when upgrading your Linux distribution.

Instead, follow the Isso docs by creating a virtualenv and then run pip install isso and pip install gunicorn from within the virtualenv. Then, when creating a systemd service, make sure to point to the gunicorn executable in that virtualenv (e.g.
/opt/isso/bin/gunicorn). It should load and run Isso from the same virtualenv.

Making a Mailing List for a Jekyll Blog Using Sendy

Wed, 30 Aug 2017 00:00:00 +0000

When my beloved Google Reader was discontinued in 2013, I stopped regularly checking RSS feeds. Apparently, I am not alone. It seems like there’s a new article every month arguing either that RSS is dead or RSS is not dead yet. Maybe RSS will stick around to serve as a cross-site communication backbone, but I don’t think anyone will refute that RSS feeds are declining in consumer use. Facebook, Twitter, and other aggregators are where people really go. However, I noticed that I still follow some small infrequent blogs through mailing lists that they offer. I’m really happy to see an email sign up on blogs I like, because it means I’ll know when they post new content in the future. I check my email regularly unlike my RSS feeds.

Even though I’m sure my blog is still too uninteresting and unheard of to get many signups, I still wanted to know what it took to make a blog mailing list. RSS is super simple for website owners, because all they need to do is dump all of their content into a specially formatted XML file, host it, and let RSS readers deal with all the complexity. In my blog, I didn’t even need a Jekyll plugin. Email is significantly more difficult. With email, the website owner owns more of the complexity. And, spam filters make it unfeasible to roll your own email server. A couple people can mark you as spam, and BAM: now you are blacklisted and you have to move to a new IP address. This is why most people turn to a hosted service like Mailchimp. Though, I was dissatisfied with that because of the high costs and measly free tier.

Amazon Simple Email Service (SES) deals with all the complexity of email for you and is also cheap. In fact, it’s free unless you have more than 62,000 subscribers or post way more than around once a month, and even after that it’s a dime for every 1,000 emails sent. Frankly, no one can really compete with what Amazon is offering here.

Okay, so that covers sending the emails, but what about collecting and storing subscriptions? SES doesn’t handle any of that. I searched around a long time for something simple and free that wouldn’t require me setting up a server ¹. I eventually ended up going with Sendy because it looked like a well-designed product exactly for this use case that also handled drafting emails, email templates, confirmation emails, and analytics. It costs a one-time fee of $59 and I was willing to fork that over for quality software. Especially since most other email newsletter services require some sort of monthly subscription that scales with the number of emails you are sending.

Unfortunately, since Sendy is self-hosted, I had to add a dynamic server to my otherwise completely static Jekyll website hosted for free on Github Pages. You can put Sendy on pretty much anything that runs PHP and MySQL including the cheap t2.micro Amazon EC2 instance type. If you are clever, you might find a cheaper way. I already had a t2.medium for general development, tinkering, and hosting, so I just used that.

There are many guides out there for setting up MySQL and Apache, so I won’t go over that. But, I do want to mention how I got Sendy to integrate with nginx which is the server engine I was already using. I like to put separate services I’m running under different subdomains of my domain hallada.net even though they are running on the same server and IP address. For Sendy, I chose list.hallada.net ². Setting up another subdomain in nginx requires creating a new server block. There’s a great Gist of a config for powering Sendy using nginx and FastCGI, but I ran into so many issues with the subdomain that I decided to use nginx as a proxy to the Apache mod_php site running Sendy. I’ll just post my config here:

server {
    listen 80;
    listen [::]:80;

    server_name list.hallada.net;

    root /var/www/html/sendy;
    index index.php;

    location /l/ {
        rewrite ^/l/([a-zA-Z0-9/]+)$ /l.php?i=$1 last;
    }

    location /t/ {
        rewrite ^/t/([a-zA-Z0-9/]+)$ /t.php?i=$1 last;
    }

    location /w/ {
        rewrite ^/w/([a-zA-Z0-9/]+)$ /w.php?i=$1 last;
    }

    location /unsubscribe/ {
        rewrite ^/unsubscribe/(.*)$ /unsubscribe.php?i=$1 last;
    }

    location /subscribe/ {
        rewrite ^/subscribe/(.*)$ /subscribe.php?i=$1 last;
    }

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $host;
        proxy_pass http://127.0.0.1:8080/sendy/;
    }
}

Basically, this proxies all of the requests through to Apache which I configured to run on port 8080 by changing the Listen directive in /etc/apache2/ports.conf.

I also had to add RewriteBase /sendy to the end of the .htcaccess file in the sendy directory (which, for me, was in /var/www/html/sendy). This basically forces Sendy to use urls that start with http://list.hallada.net instead of http://list.hallada.net/sendy which I thought was redundant since I am dedicating the whole subdomain to sendy.

A perplexing issue I ran into was that Gmail accounts were completely dropping (not even bouncing!) any emails I sent to them if I used my personal email [email protected] as the from address. I switched to [email protected] for the from address and emails went through fine after that ³. The issue seems unresolved as of this post.

Lastly, I needed to create a form on my website for readers to sign up for the mailing list. Sendy provides the HTML in the UI to create the form, which I tweaked a little and placed in a Jekyll includes template partial that I could include on both the post layout and the blog index template. I refuse to pollute the internet with yet another annoying email newsletter form that pops up while you are trying to read the article, so you can find my current version at the bottom of this article where it belongs ⁴.

All in all, setting up a mailing list this way wasn’t too bad except for the part where I spent way too much time fiddling with nginx configs. But, I always do that, so I guess that’s expected.

As for the content of the newsletter, I haven’t figured out how to post the entirety of a blog post into the HTML format of an email as soon as I commit a new post yet. So, I think for now I will just manually create a new email campaign in Sendy (from an email template) that will have a link to the new post, and send that.

It would be interesting to look into creating a Google Form that submits rows to a Google Sheet and then triggering a AWS Lambda service that iterates over the rows using something like the Google Sheets Python API and sending an email for every user using the Amazon SES API (python-amazon-ses-api might also be useful there). ↩
I ran into a hiccup verifying this domain for Amazon SES using the Namecheap advanced DNS settings because it only allowed me to set up one MX record, but I already had one for my root hallada.net domain that I needed. So, I moved to Amazon’s Route 53 instead ⁵ which made setting up the DKIM verification really easy since Amazon SES gave a button to create the necessary DNS records directly in my Route 53 account. ↩
Obviously a conspiracy by Google to force domination of Gmail. ↩
Yes, I really hate those pop-ups. ↩
As Amazon continues its plan for world domination it appears I’m moving more and more of my personal infrastructure over to Amazon as well… ↩

Proximity Structures: Playing around with PixiJS

Mon, 07 Aug 2017 00:00:00 +0000

I’ve been messing around with a library called PixiJS which allows you to create WebGL animations which will fall back to HTML5 canvas if WebGL is not available in the browser. I mostly like it because the API is similar to HTML5 canvas which I was already familiar with. I can’t say that I like the PixiJS API and documentation that much, though. For this project, I mostly just used a small portion of it to create WebGL (GPU accelerated) primitive shapes (lines and circles).

Play with it here: http://proximity.hallada.net

Read/clone the code here: https://github.com/thallada/proximity-structures

The idea was inspired by all those countless node network graphics that I see all the time as stock graphics on generic tech articles.

This was really fun to program. I didn’t care much about perfect code, I just kept hacking one thing onto another while watching the instantaneous feedback of the points and lines responding to my changes until I had something worth sharing.

Details

The majority of the animation you see is based on tweening. Each point has an origin and destination stored in memory. Every clock tick (orchestrated by the almighty requestAnimationFrame), the main loop calculates where each point should be in the path between its origin and destination based on how long until it completes its “cycle”. There is a global cycleDuration, defaulted to 60. Every frame increments the cycle counter by 1 until it reaches 60, at which point it folds over back to 0. Every point is assigned a number between 1 and 60. This is its start cycle. When the global cycle counter equals a point’s start cycle number, the point has reached its destination and a new target destination is randomly chosen.

Each point is also randomly assigned a color. When a point is within connectionDistance of another point in the canvas, a line is drawn between the two points, their colors are averaged, and the points’ colors become the average color weighted by the distance between the points. You can see clusters of points converging on a color in the animation.

Click interaction is implemented by modifying point target destinations within a radius around the click. Initially, a mouse hover will push points away. Clicking and holding will draw points in, progressively growing the effect radius in the process to capture more and more points.

I thought it was really neat that without integrating any physics engine whatsoever, I ended up with something that looked sort of physics based thanks to the tweening functions. Changing the tweening functions that the points use seems to change the physical properties and interactions of the points. The elastic tweening function makes the connections between the points snap like rubber bands. And, while I am not drawing any explicit polygons, just points and lines based on proximity, it sometimes looks like the points are coalescing into some three-dimensional structure.

I’ll probably make another procedural animation like this in the future since it was so fun. Next time, I’ll probably start from the get-go in ES2015 (or ES7, or ES8??) and proper data structures.

Generating Random Poems with Python

Tue, 11 Jul 2017 00:00:00 +0000

In this post, I will demonstrate how to generate random text using a few lines of standard python and then progressively refine the output until it looks poem-like.

If you would like to follow along with this post and run the code snippets yourself, you can clone my NLP repository and run the Jupyter notebook.

You might not realize it, but you probably use an app everyday that can generate random text that sounds like you: your phone keyboard.

Just by tapping the next suggested word over and over, you can generate text. So how does it work?

Corpus

First, we need a corpus: the text our generator will recombine into new sentences. In the case of your phone keyboard, this is all the text you’ve ever typed into your keyboard. For our example, let’s just start with one sentence:

corpus = 'The quick brown fox jumps over the lazy dog'

Tokenization

Now we need to split this corpus into individual tokens that we can operate on. Since our objective is to eventually predict the next word from the previous word, we will want our tokens to be individual words. This process is called tokenization. The simplest way to tokenize a sentence into words is to split on spaces:

words = corpus.split(' ')
words

['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']

Bigrams

Now, we will want to create bigrams. A bigram is a pair of two words that are in the order they appear in the corpus. To create bigrams, we will iterate through the list of the words with two indices, one of which is offset by one:

bigrams = [b for b in zip(words[:-1], words[1:])]
bigrams

[('The', 'quick'),
 ('quick', 'brown'),
 ('brown', 'fox'),
 ('fox', 'jumps'),
 ('jumps', 'over'),
 ('over', 'the'),
 ('the', 'lazy'),
 ('lazy', 'dog')]

How do we use the bigrams to predict the next word given the first word?

Return every second element where the first element matches the condition:

condition = 'the'
next_words = [bigram[1] for bigram in bigrams
          if bigram[0].lower() == condition]
next_words

['quick', 'lazy']

We have now found all of the possible words that can follow the condition “the” according to our corpus: “quick” and “lazy”.

(The quick) (quick brown) ... (the lazy) (lazy dog)

Either “quick” or “lazy” could be the next word.

Trigrams and N-grams

We can partition our corpus into groups of threes too:

(The quick brown) (quick brown fox) ... (the lazy dog)

Or, the condition can be two words (condition = 'the lazy'):

(The quick brown) (quick brown fox) ... (the lazy dog)

These are called trigrams.

We can partition any N number of words together as n-grams.

Conditional Frequency Distributions

Earlier, we were able to compute the list of possible words to follow a condition:

next_words

['quick', 'lazy']

But, in order to predict the next word, what we really want to compute is what is the most likely next word out of all of the possible next words. In other words, find the word that occurred the most often after the condition in the corpus.

We can use a Conditional Frequency Distribution (CFD) to figure that out! A CFD can tell us: given a condition, what is likelihood of each possible outcome.

This is an example of a CFD with two conditions, displayed in table form. It is counting words appearing in a text collection (source: nltk.org).

Let’s change up our corpus a little to better demonstrate the CFD:

words = ('The quick brown fox jumped over the '
         'lazy dog and the quick cat').split(' ')
print words

['The', 'quick', 'brown', 'fox', 'jumped', 'over', 'the', 'lazy', 'dog', 'and', 'the', 'quick', 'cat']

Now, let’s build the CFD. I use defaultdicts to avoid having to initialize every new dict.

from collections import defaultdict
cfd = defaultdict(lambda: defaultdict(lambda: 0))
for i in range(len(words) - 2):  # loop to the next-to-last word
    cfd[words[i].lower()][words[i+1].lower()] += 1

# pretty print the defaultdict
{k: dict(v) for k, v in dict(cfd).items()}

{'and': {'the': 1},
 'brown': {'fox': 1},
 'dog': {'and': 1},
 'fox': {'jumped': 1},
 'jumped': {'over': 1},
 'lazy': {'dog': 1},
 'over': {'the': 1},
 'quick': {'brown': 1},
 'the': {'lazy': 1, 'quick': 2}}

So, what’s the most likely word to follow 'the'?

max(cfd['the'])

'quick'

Whole sentences can be the conditions and values too. Which is basically the way cleverbot works.

Random Text

Lets put this all together, and with a little help from nltk generate some random text.

import nltk
import random

TEXT = nltk.corpus.gutenberg.words('austen-emma.txt')

# NLTK shortcuts :)
bigrams = nltk.bigrams(TEXT)
cfd = nltk.ConditionalFreqDist(bigrams)

# pick a random word from the corpus to start with
word = random.choice(TEXT)
# generate 15 more words
for i in range(15):
    print word,
    if word in cfd:
        word = random.choice(cfd[word].keys())
    else:
        break

Which outputs something like:

her reserve and concealment towards some feelings in moving slowly together .
You will shew

Great! This is basically what the phone keyboard suggestions are doing. Now how do we take this to the next level and generate text that looks like a poem?

Random Poems

Generating random poems is accomplished by limiting the choice of the next word by some constraint:

words that rhyme with the previous line
words that match a certain syllable count
words that alliterate with words on the same line
etc.

Rhyming

Written English != Spoken English

English has a highly nonphonemic orthography, meaning that the letters often have no correspondence to the pronunciation. E.g.:

“meet” vs. “meat”

The vowels are spelled differently, yet they rhyme ¹.

So if the spelling of the words is useless in telling us if two words rhyme, what can we use instead?

International Phonetic Alphabet (IPA)

The IPA is an alphabet that can represent all varieties of human pronunciation.

meet: /mit/
meat: /mit/

Note that this is only the IPA transcription for only one accent of English. Some English speakers may pronounce these words differently which could be represented by a different IPA transcription.

Syllables

How can we determine the number of syllables in a word? Let’s consider the two words “poet” and “does”:

“poet” = 2 syllables
“does” = 1 syllable

The vowels in these two words are written the same, but are pronounced differently with a different number of syllables.

Can the IPA tell us the number of syllables in a word too?

poet: /ˈpoʊət/
does: /ˈdʌz/

Not really… We cannot easily identify the number of syllables from those transcriptions. Sometimes the transcriber denotes syllable breaks with a . or a ', but sometimes they don’t.

Arpabet

The Arpabet is a phonetic alphabet developed by ARPA in the 70s that:

Encodes phonemes specific to American English.
Meant to be a machine readable code. It is ASCII only.
Denotes how stressed every vowel is from 0-2.

This is perfect! Because of that third bullet, a word’s syllable count equals the number of digits in the Arpabet encoding.

CMU Pronouncing Dictionary (CMUdict)

A large open source dictionary of English words to North American pronunciations in Arpanet encoding. Conveniently, it is also in NLTK…

Counting Syllables

import string
from nltk.corpus import cmudict
cmu = cmudict.dict()

def count_syllables(word):
    lower_word = word.lower()
    if lower_word in cmu:
        return max([len([y for y in x if y[-1] in string.digits])
                    for x in cmu[lower_word]])

print("poet: {}\ndoes: {}".format(count_syllables("poet"),
                                  count_syllables("does")))

Results in:

poet: 2
does: 1

Buzzfeed Haiku Generator

To see this in action, try out a haiku generator I created that uses Buzzfeed article titles as a corpus. It does not incorporate rhyming, it just counts the syllables to make sure it’s 5-7-5. You can view the full code here.

Run it live at: http://mule.hallada.net/nlp/buzzfeed-haiku-generator/

Syntax-aware Generation

Remember these?

Mad Libs worked so well because they forced the random words (chosen by the players) to fit into the syntactical structure and parts-of-speech of an existing sentence.

You end up with syntactically correct sentences that are semantically random. We can do the same thing!

NLTK Syntax Trees!

NLTK can parse any sentence into a syntax tree. We can utilize this syntax tree during poetry generation.

from stat_parser import Parser
parsed = Parser().parse('The quick brown fox jumps over the lazy dog.')
print parsed

Syntax tree output as an s-expression:

(S
  (NP (DT the) (NN quick))
  (VP
    (VB brown)
    (NP
      (NP (JJ fox) (NN jumps))
      (PP (IN over) (NP (DT the) (JJ lazy) (NN dog)))))
  (. .))

parsed.pretty_print()

And the same tree visually pretty printed in ASCII:

                              S                            
      ________________________|__________________________   
     |               VP                                  | 
     |           ____|_____________                      |  
     |          |                  NP                    | 
     |          |         _________|________             |  
     |          |        |                  PP           | 
     |          |        |          ________|___         |  
     NP         |        NP        |            NP       | 
  ___|____      |     ___|____     |     _______|____    |  
 DT       NN    VB   JJ       NN   IN   DT      JJ   NN  . 
 |        |     |    |        |    |    |       |    |   |  
the     quick brown fox     jumps over the     lazy dog  . 

NLTK also performs part-of-speech tagging on the input sentence and outputs the tag at each node in the tree. Here’s what each of those mean:

S	Sentence
VP	Verb Phrase
NP	Noun Phrase
DT	Determiner
NN	Noun (common, singular)
VB	Verb (base form)
JJ	Adjective (or numeral, ordinal)
.	Punctuation

Now, let’s use this information to swap matching syntax sub-trees between two corpora (source for the generate function).

from syntax_aware_generate import generate

# inserts matching syntax subtrees from trump.txt into
# trees from austen-emma.txt
generate('trump.txt', word_limit=10)

(SBARQ
  (SQ
    (NP (PRP I))
    (VP (VBP do) (RB not) (VB advise) (NP (DT the) (NN custard))))
  (. .))
I do not advise the custard .
==============================
I do n't want the drone !
(SBARQ
  (SQ
    (NP (PRP I))
    (VP (VBP do) (RB n't) (VB want) (NP (DT the) (NN drone))))
  (. !))

Above the line is a sentence selected from a corpus of Jane Austen’s Emma. Below it is a sentence generated by walking down the syntax tree and finding sub-trees from a corpus of Trump’s tweets that match the same syntactical structure and then swapping the words in.

The result can sometimes be amusing, but more often than not, this approach doesn’t fare much better than the n-gram based generation.

spaCy

I’m only beginning to experiment with the spaCy Python library, but I like it a lot. For one, it is much, much faster than NLTK:

https://spacy.io/docs/api/#speed-comparison

The API takes a little getting used to coming from NLTK. It doesn’t seem to have any sort of out-of-the-box solution to printing out syntax trees like above, but it does do part-of-speech tagging and dependency relation mapping which should accomplish about the same. You can see both of these visually with displaCy.

Neural Network Based Generation

If you haven’t heard all the buzz about neural networks, they are a particular technique for machine learning that’s inspired by our understanding of the human brain. They are structured into layers of nodes which have connections to other nodes in other layers of the network. These connections have weights which each node multiplies by the corresponding input and enters into a particular activation function to output a single number. The optimal weights for solving a particular problem with the network are learned by training the network using backpropagation to perform gradient descent on a particular cost function that tries to balance getting the correct answer while also generalizing the network enough to perform well on data the network hasn’t seen before.

Long short-term memory (LSTM) is a type of recurrent neural network (RNN) (a network with cycles) that can remember previous values for a short or long period of time. This property makes them remarkably effective at a multitude of tasks, one of which is predicting text that will follow a given sequence. We can use this to continually generate text by inputting a seed, appending the generated output to the end of the seed, removing the first element from the beginning of the seed, and then inputting the seed again, following the same process until we’ve generated enough text from the network (paper on using RNNs to generate text).

Luckily, a lot of smart people have done most of the legwork so you can just download their neural network architecture and train it yourself. There’s char-rnn which has some really exciting results for generating texts (e.g. fake Shakespeare). There’s also word-rnn which is a modified version of char-rnn that operates on words as a unit instead of characters. Follow my last blog post on how to install TensorFlow on Ubuntu 16.04 and you’ll be almost ready to run a TensorFlow port of word-rnn: word-rnn-tensorflow.

I plan on playing around with NNs a lot more to see what kind of poetry-looking text I can generate from them.

Fun fact: They used to be pronounced differently in Middle English during the invention of the printing press and standardized spelling. The Great Vowel Shift happened after, and is why they are now pronounced the same. ↩

How to Install TensorFlow on Ubuntu 16.04 with GPU Support

Tue, 20 Jun 2017 00:00:00 +0000

I found the tensorflow documentation rather lacking for installation instructions, especially in regards to getting GPU support. I’m going to write down my notes from wrangling with the installation here for future reference and hopefully this helps someone else too.

This will invariably go out-of-date at some point, so be mindful of the publish date of this post. Make sure to cross-reference other documentation that has more up-to-date information.

Assumptions

These instructions are very specific to my environment, so this is what I am assuming:

You are running Ubuntu 16.04. (I have 16.04.1)
- You can check this in the output of uname -a
You have a 64 bit machine.
- You can check this with uname -m. (should say x86_64)
You have an NVIDIA GPU that has CUDA Compute Capability 3.0 or higher. NVIDIA documentation has a full table of cards and their Compute Capabilities. (I have a GeForce GTX 980 Ti)
- You can check what card you have in Settings > Details under the label “Graphics”
- You can also check by verifying there is any output when you run lspci | grep -i nvidia
You have a linux kernel version 4.4.0 or higher. (I have 4.8.0)
- You can check this by running uname -r
You have gcc version 5.3.1 or higher installed. (I have 5.4.0)
- You can check this by running gcc --version
You have the latest proprietary NVIDIA drivers installed.
- You can check this and install it if you haven’t in the “Additional Drivers” tab in the “Software & Updates” application (update-manager). (I have version 375.66 installed)
You have the kernel headers installed.
- Just run sudo apt-get install linux-headers-$(uname -r) to install them if you don’t have them installed already.
You have Python installed. The exact version shouldn’t matter, but for the rest of this post I’m going to assume you have python3 installed.
- You can install python3 by running sudo apt-get install python3. This will install Python 3.5.
- Bonus points: you can install Python 3.6 by following this answer, but Python 3.5 should be fine.

Install the CUDA Toolkit 8.0

NVIDIA has a big scary documentation page on this, but I will summarize the only the parts you need to know here.

Go to the CUDA Toolkit Download page. Click Linux > x86_64 > Ubuntu > 16.04 > deb (network).

Click download and then follow the instructions, copied here:

sudo dpkg -i cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
sudo apt-get update
sudo apt-get install cuda

This will install CUDA 8.0. It installed it to the directory /usr/local/cuda-8.0/ on my machine.

There are some post-install actions we must follow:

Edit your ~/.bashrc
- Use your favorite editor gedit ~/.bashrc, nano ~/.bashrc, vim ~/.bashrc, whatever.

Add the following lines to the end of the file:

# CUDA 8.0 (nvidia) paths
export CUDA_HOME=/usr/local/cuda-8.0
export PATH=/usr/local/cuda-8.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Save and exit.
Run source ~/.bashrc.
Install writable samples by running the script cuda-install-samples-8.0.sh ~/.
- If the script cannot be found, the above steps didn’t work :(
- I don’t actually know if the samples are absolutely required for what I’m using CUDA for, but it’s recommended according to NVIDIA, and compiling them will output a nifty deviceQuery binary which can be ran to test if everything is working properly.
Make sure nvcc -V outputs something.
- If an error, the above steps 1-4 didn’t work :(
cd ~/NVIDIA_CUDA-8.0_Samples, cross your fingers, and run make
- The compile will take a while
- My compile actually errored near the end with an error about /usr/bin/ld: cannot find -lnvcuvid I think that doesn’t really matter because the binary files were still output.
Try running ~/NVIDIA_CUDA-8.0_Samples/bin/x86_64/linux/release/deviceQuery to see if you get any output. Hopefully you will see your GPU listed.

Install cuDNN v5.1

This AskUbuntu answer has good instructions. Here are the instructions specific to this set-up:

Visit the NVIDIA cuDNN page and click “Download”.
Join the program and fill out the survey.
Agree to the terms of service.
Click the link for “Download cuDNN v5.1 (Jan 20, 2017), for CUDA 8.0”
Download the “cuDNN v5.1 Library for Linux” (3rd link from the top).

Untar the downloaded file. E.g.:

cd ~/Downloads
tar -xvf cudnn-8.0-linux-x64-v5.1.tgz

Install the cuDNN files to the CUDA folder:

cd cuda
sudo cp -P include/* /usr/local/cuda-8.0/include/
sudo cp -P lib64/* /usr/local/cuda-8.0/lib64/
sudo chmod a+r /usr/local/cuda-8.0/lib64/libcudnn*

Install libcupti-dev

This one is simple. Just run:

sudo apt-get install libcupti-dev

Create a Virtualenv

I recommend using virtualenvwrapper to create the tensorflow virtualenv, but the TensorFlow docs still have instructions to create the virtualenv manually.

Install virtualenvwrapper. Make sure to add the required lines to your ~/.bashrc.

Create the virtualenv:

mkvirtualenv --python=python3 tensorflow

Install the TensorFlow with GPU support

If you just run pip install tensorflow you will not get GPU support. To install the correct version you will have to install from a particular url. Here is the install command you will have to run to install TensorFlow 1.2 for Python 3.5 with GPU support:

pip install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.2.0-cp35-cp35m-linux_x86_64.whl

If you need a different version of TensorFlow, you can edit the version number in the URL. Same with the Python version (change cp35 to cp36 to install for Python 3.6 instead, for example).

Test that the installation worked

Save this script from the TensorFlow tutorials to a file called test_gpu.py:

# Creates a graph.
with tf.device('/cpu:0'):
  a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
  b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
# Runs the op.
print(sess.run(c))

And then run it:

python test_gpu.py

You should see your GPU card listed under “Device mapping:” and that each task in the compute graph is assigned to gpu:0.

If you see “Device mapping: no known devices” then something went wrong and TensorFlow cannot access your GPU.

Generating Realistic Satellite Imagery with Deep Neural Networks

Wed, 06 Jan 2016 00:00:00 +0000

I’ve been doing a lot of experimenting with neural-style the last month. I think I’ve discovered a few exciting applications of the technique that I haven’t seen anyone else do yet. The true power of this algorithm really shines when you can see concrete examples.

Skip to the Applications part of this post to see the outputs from my experimentation if you are already familiar with DeepDream, Deep Style, and all the other latest happenings in generating images with deep neural networks.

Background and History

On May 18, 2015 at 2 a.m., Alexander Mordvintsev, an engineer at Google, did something with deep neural networks that no one had done before. He took a net designed for recognizing objects in images and used it to generate objects in images. In a sense, he was telling these systems that mimic the human visual cortex to hallucinate things that weren’t really there. The results looked remarkably like LSD trips or what a schizophrenic person sees on a blank wall.

Mordvintsev’s discovery quickly gathered attention at Google once he posted images from his experimentation on the company’s internal network. On June 17, 2015, Google posted a blog post about the technique (dubbed “Inceptionism”) and how it was useful for opening up the notoriously black-boxed neural networks using visualizations that researchers could examine. These machine hallucinations were key for identifying the features of objects that neural networks used to tell one object from another (like a dog from a cat). But the post also revealed the beautiful results of applying the algorithm iteratively on it’s own outputs and zooming out at each step.

The internet exploded in response to this post. And once Google posted the code for performing the technique, people began experimenting and sharing their fantastic and creepy images with the world.

Then, on August, 26, 2015, a paper titled “A Neural Algorithm of Artistic Style” was published. It showed how one could identify which layers of deep neural networks recognized stylistic information of an image (and not the content) and then use this stylistic information in Google’s Inceptionism technique to paint other images in the style of any artist. A few implementations of the paper were put up on Github. This exploded the internet again in a frenzy. This time, the images produced were less like psychedelic-induced nightmares but more like the next generation of Instagram filters (reddit how-to).

People began to wonder what all of this meant to the future of art. Some of the results produced where indistinguishable from the style of dead artists’ works. Was this a demonstration of creativity in computers or just a neat trick?

On November, 19, 2015, another paper was released that demonstrated a technique for generating scenes from convolutional neural nets (implementation on Github). The program could generate random (and very realistic) bedroom images from a neural net trained on bedroom images. Amazingly, it could also generate the same bedroom from any angle. It could also produce images of the same procedurally generated face from any angle. Theoretically, we could use this technology to create procedurally generated game art.

The main thing holding this technology back from revolutionizing procedurally generated video games is that it is not real-time. Using neural-style to apply artistic style to a 512 by 512 pixel content image could take minutes even on the top-of-the-line GTX Titan X graphics card. Still, I believe this technology has a lot of potential for generating game art even if it can’t act as a real-time filter.

Applications: Generating Satellite Images for Procedural World Maps

I personally know very little machine learning, but I have been able to produce a lot of interesting results by using the tool provided by neural-style.

Inspired by Kaelan’s procedurally generated world maps, I wanted to extend the idea by generating realistic satellite images of the terrain maps. The procedure is simple: take a generated terrain map and apply the style of a real-world satellite image on it using neural-style.

The generated output takes on whatever terrain is in the satellite image. Here is an output processing one of Kaelan’s maps with a arctic satellite image:

And again, with one of Kaelan’s desert maps and a satellite image of a desert:

It even works with Kaelan’s generated hexagon maps. Here’s an island hexagon map plus a satellite image of a volcanic island:

This image even produced an interesting three-dimensional effect because of the volcano in the satellite image.

By the way, this also works with minecraft maps. Here’s a minecraft map I found on the internet plus a satellite image from Google Earth:

No fancy texture packs or 3-D rendering needed :).

Here is the Fallout 4 grayscale map plus a satellite image of Boston:

Unfortunately, it puts the built-up dense part of the city in the wrong part of the geographic area. But, this is understandable since we gave the algorithm no information on where that is on the map.

We can also make the generated terrain maps look like old hand-drawn maps using neural-style. With Kaelan’s terrain map as the content and the in-game Elder Scrolls IV Oblivion map of Cyrodiil as the style we get this:

It looks cool, but the water isn’t conveyed very clearly (e.g. makes deep water look like land). Neural-style seems to work better when there is lots of color in both images.

Here is the output of the hex terrain plus satellite map above and the Cyrodiil map which looks a little cleaner:

I was interested to see what neural-style could generate from random noise, so I rendered some clouds in GIMP and ran it with a satellite image of Mexico City from Google Earth (by the way, I’ve been getting high quality Google Earth shots from earthview.withgoogle.com).

Not bad for a neural net without a degree in urban planning.

I also tried generating on random noise with a satellite image of a water treatment plant in Peru

Applications: More Fun

For fun, here are some other outputs that I liked.

My photo of Boston’s skyline as the content and Vincent van Gogh’s The Starry Night as the style:

A photo of me (by Aidan Bevacqua) and Forrest in the end of Autumn by Caspar David Friedrich:

Another photo of me by Aidan in the same style:

A photo of me on a mountain (by Aidan Bevacqua) and pixel art by Paul Robertson

A photo of a park in Copenhagen I took and a painting similar in composition, Avenue of Poplars at Sunset by Vincent van Gogh:

My photo of the Shenandoah National Park and this halo graphic from GMUNK (GMUNK):

A photo of me by Aidan and a stained glass fractal:

Same photo of me and some psychedelic art by GMUNK

New York City and a rainforest:

Kowloon Walled City and a National Geographic Map:

A photo of me by Aidan and Head of Lioness by Theodore Gericault:

Photo I took of a Norwegian forest and The Mountain Brook by Albert Bierstadt:

Limitations

I don’t have infinite money for a GTX Titan X, so I’m stuck with using OpenCL on my more-than-a-few-generations-old AMD card. It takes about a half-hour to generate one 512x512 px image in my set-up (which makes the feedback loop for correcting mistakes very long). And sometimes the neural-style refuses to run on my GPU (I suspect it runs out of VRAM), so I have to run it on my CPU which takes even longer…

I am unable to generate bigger images (though the author has been able to generate up to 1920x1010 px). As the size of the output increases the amount of memory and time to generate also increases. And, it’s not practical to just generate thumbnails to test parameters, because increasing the image size will probably generate a very different image since all the other parameters stay the same even though they are dependent on the image size.

Some people have had success running these neural nets on GPU spot instances in AWS. It would be certainly cheaper than buying a new GPU in the short-term.

So, I have a few more ideas for what to run, but it will take me quite a while to get through the queue.