In the last part, I gave a brief glimpse into the graphics part of implementing a custom game engine on iOS — setting a Metal-backed view as the main view of the game’s View Controller. This view is what will contain the entire visual representation of the game. In this part, I will detail how I go about setting up the Metal view and how the game itself renders a frame into it. And just to reiterate what I said in the last part, this is just one way of doing this; this works for me and the game I am making. If nothing else, let it serve as a guide, reference, and/or inspiration.
First of all, I make use of a software renderer that I wrote for my game. This is quite unusual these days with all the high-powered and specialized GPUs in nearly every device out there, but you might be surprised at what a properly optimized software renderer is capable of; not to mention some nice benefits as well, such as avoiding the complexities of GPU APIs and porting issues (Metal vs. DirectX vs. OpenGL, etc.). I also really enjoyed the knowledge and insight I gained through the process of writing my own renderer. So that being said, why the need for the Metal view, or the use of Metal at all on iOS?
The general overview of the way it works is that each frame of the game is simulated and then rendered into a bitmap by the game layer. This bitmap (or more specifically the memory for this bitmap) is provided to the game by the platform layer. Once the game is done simulating a frame and rendering the result into this bitmap, the platform layer takes this bitmap and displays it on the screen using Metal.
To represent this bitmap, I have a struct called PlatformTexture defined in the bridging header file we saw in the last part:
struct PlatformTexture {
uint16_t width;
uint16_t height;
size_t bytesPerRow;
uint32_t *memory;
uint32_t *texture;
};
struct PlatformTexture* ios_create_platform_texture(uint16_t screenWidth, uint16_t screenHeight);
void ios_graphics_initialize(struct PlatformTexture *platformTexture);
void ios_graphics_deinitialize(struct PlatformTexture *platformTexture);
The implementations of these functions (in the bridging .cpp file) look like this:
struct PlatformTexture*
ios_create_platform_texture(uint16_t screenWidth, uint16_t screenHeight) {
static PlatformTexture platformTexture = {};
platformTexture.width = screenWidth;
platformTexture.height = screenHeight;
return &platformTexture;
}
typedef uint32_t bbPixel;
constexpr size_t kPixelSize = sizeof(bbPixel);
void
ios_graphics_initialize(struct PlatformTexture *platformTexture) {
uint16_t allocWidth = platformTexture->width + 2*CLIP_REGION_PIXELS;
uint16_t allocHeight = platformTexture->height + 2*CLIP_REGION_PIXELS;
size_t memorySize = allocWidth * allocHeight * kPixelSize;
platformTexture->memory = (bbPixel *)ios_allocate_memory(memorySize);
if (platformTexture->memory) {
platformTexture->texture = platformTexture->memory + (intptr_t)(CLIP_REGION_PIXELS * allocWidth + CLIP_REGION_PIXELS);
platformTexture->bytesPerRow = platformTexture->width * kPixelSize;
}
}
void
ios_graphics_deinitialize(struct PlatformTexture *platformTexture) {
ios_deallocate_memory(platformTexture->memory);
}
The instance itself of PlatformTexture is owned by the bridging .cpp file, so it returns a pointer to the caller (in this case the platform layer). Initialization allocates memory for the texture, including extra for a clip region, or pixel “apron”, around the texture to guard against writing past the bounds of the texture. (It’s also useful for effects like screen shake.) Essentially it can be visualized like this:
Visualization of the PlatformTexture memory.
Furthermore, the pixel format of the texture is the standard RGBA format with each pixel represented as a packed 32-bit integer (8 bits per component).
Initialization of the PlatformTexture is handled within the ViewController class:
override func viewDidLoad() {
super.viewDidLoad()
if let ptr = ios_create_platform_texture(UInt16(view.frame.width), UInt16(view.frame.height)) {
platformTexture = ptr.pointee
ios_graphics_initialize(&platformTexture)
}
startGame()
}
deinit {
ios_graphics_deinitialize(&platformTexture)
}
To simulate a frame of the game, the update_and_render function from the game layer needs to be called every 1/60th of a second (to achieve 60 fps), and as I mentioned earlier, the platform layer needs to pass the game layer some memory for the bitmap that will be rendered into. This memory is, of course, the texture pointer in the PlatformTexture type. Here is the function as included in the .h/.cpp bridging files:
// declaration in .h file
void ios_update_and_render(struct PlatformTexture *platformTexture, float dt);
// definition in .cpp file
void
ios_update_and_render(PlatformTexture *platformTexture, float dt) {
// Map input (from previous part)
bbTextureBuffer buf = {0};
buf.width = platformTexture->width;
buf.height = platformTexture->height;
buf.buffer = platformTexture->texture;
update_and_render(&buf, &input, &gameMemory, &platform, dt);
}
The call to the ios_update_and_render function converts the PlatformTexture into the struct the game actually expects, and then makes a call to the game layer to update and render the frame. (Again, this conversion is needed because bbTextureBuffer is declared in a C++ interface file which cannot interoperate with Swift, so the plain C PlatformTexture data type acts as a bridge between the Swift side and the C++ side.)
The ios_update_and_render function is called from the getFrame method of the View Controller (which was shown in the last part on setting up the platform layer):
@objc private func getFrame(_ sender: CADisplayLink) {
ios_begin_frame()
let dt = Float(sender.targetTimestamp - sender.timestamp)
ios_update_and_render(&platformTexture, dt)
if let metalView = view as? MetalView {
metalView.renderFrame(platformTexture: platformTexture)
}
ios_end_frame()
}
Here we see how Metal comes into the picture (no pun intended). After the game is done simulating and rendering the frame into the PlatformTexture object, the Metal view takes over and draws the image to the screen.
We saw in the previous part on setting up the platform layer where the MetalView got initialized, but now let’s look at what that contains. UIViews in iOS are all what Apple calls “layer-backed”, containing the backing Core Animation layer that defines the actual visual contents of the view (unlike NSView on macOS which need to be assigned a layer if it is to draw or display something). To make a UIView subclass a Metal-backed view, we need to tell it to use the CAMetalLayer class for it’s Core Animation layer by overwriting the class property layerClass:
class MetalView: UIView {
var commandQueue: MTLCommandQueue?
var renderPipeline: MTLRenderPipelineState?
var renderPassDescriptor = MTLRenderPassDescriptor()
var vertexBuffer: MTLBuffer?
var uvBuffer: MTLBuffer?
var texture: MTLTexture?
let semaphore = DispatchSemaphore(value: 1)
class override var layerClass: AnyClass {
return CAMetalLayer.self
}
init?(metalDevice: MTLDevice) {
super.init(frame: UIScreen.main.bounds)
guard let metalLayer = layer as? CAMetalLayer else { return nil }
metalLayer.framebufferOnly = true
metalLayer.pixelFormat = .bgra8Unorm
metalLayer.device = metalDevice
// Additional initialization...
}
}
The init method of the view configures the properties of the Metal layer. Setting the framebufferOnly property to true tells Metal that this layer will only be used as a render target, allowing for some optimizations when rendering the layer for display. Framebuffer targets must use the bgra8Unorm pixel format. This format is simply an unsigned 32-bit RGBA format, but in BGRA order.
We now get into more nitty-gritty Metal stuff. In order to do most anything in Metal, we need a command queue, which gives us a command buffer, which then allows us to encode commands for Metal to perform. In order to get a command encoder from the command buffer, we need a render pass descriptor. A render pass descriptor contains a set of attachments that represent the destination, or target, of a render pass. In other words, one of the attachments in the render pass descriptor is the color attachment, which is essentially the pixel data of a render pass. The last thing we need is a render pipeline state. This object represents a particular state during a render pass, including the vertex and fragment functions. The next part of the init method of the MetalView sets up these objects:
let library = metalDevice.makeDefaultLibrary()
let vertexShader = "basic_vertex"
let fragmentShader = "texture_fragment"
let vertexProgram = library?.makeFunction(name: vertexShader)
let fragmentProgram = library?.makeFunction(name: fragmentShader)
let renderPipelineDescriptor = MTLRenderPipelineDescriptor()
renderPipelineDescriptor.vertexFunction = vertexProgram
renderPipelineDescriptor.fragmentFunction = fragmentProgram
renderPipelineDescriptor.colorAttachments[0].pixelFormat = .bgra8Unorm
commandQueue = metalDevice.makeCommandQueue()
renderPipeline = try? metalDevice.makeRenderPipelineState(descriptor: renderPipelineDescriptor)
renderPassDescriptor.colorAttachments[0].loadAction = .dontCare
renderPassDescriptor.colorAttachments[0].storeAction = .store
First, a library gives us access to the shader functions that were compiled into the application (more on those later). Since all I need Metal to do is to draw a bitmap onto the screen, I only need two simple shader functions: a basic vertex shader, and a fragment shader for texture mapping. A render pipeline descriptor is created in order to make the render pipeline state. Above we can see that the render pipeline state is configured with the two shader functions, and the pixel format for the color attachment. The command queue is simply created from the Metal Device.
The color attachment (i.e. render target for pixel data) of the render pass descriptor is configured for its load and store actions. The load action is performed at the start of a rendering pass, and can be used to clear the attachment to a specific color. Since I will be writing the entire bitmap every frame into the color attachment, there is no need to clear it beforehand. For the store action, I specify that the results of the render pass should be saved in memory to the attachment.
The next, and final, thing that needs to be set up in the init method of the MetalView are the buffer and texture objects required for rendering. I’m not going to go into the details of texture mapping, which is what the buffer and texture objects are required for, as that is just too big of a topic, so I will assume some basic knowledge of UV texture mapping going forward.
First, I define the vertices and UV coordinates of the unit quad primitive for texture mapping (the coordinate system in Metal has +x/y going to the right and up, and -x/y going to the left and down):
fileprivate let unitQuadVertices: [Float] = [
-1.0, 1.0, 1.0,
-1.0, -1.0, 1.0,
1.0, -1.0, 1.0,
-1.0, 1.0, 1.0,
1.0, -1.0, 1.0,
1.0, 1.0, 1.0
]
fileprivate let unitQuadUVCoords: [Float] = [
0.0, 0.0,
0.0, 1.0,
1.0, 1.0,
0.0, 0.0,
1.0, 1.0,
1.0, 0.0
]
As we can see, these vertices and UV coordinates define two triangles that make up the unit quad. Metal buffers then need to be created to contain this data to send to the GPU:
let vertexBufferSize = MemoryLayout<Float>.size * unitQuadVertices.count
let uvBufferSize = MemoryLayout<Float>.size * unitQuadUVCoords.count
guard let sharedVertexBuffer = metalDevice.makeBuffer(bytes: unitQuadVertices, length: vertexBufferSize, options: .storageModeShared),
let sharedUVBuffer = metalDevice.makeBuffer(bytes: unitQuadUVCoords, length: uvBufferSize, options: .storageModeShared) else {
return nil
}
vertexBuffer = metalDevice.makeBuffer(length: vertexBufferSize, options: .storageModePrivate)
uvBuffer = metalDevice.makeBuffer(length: uvBufferSize, options: .storageModePrivate)
guard let vertexBuffer = vertexBuffer, let uvBuffer = uvBuffer else {
return nil
}
let textureWidth = Int(frame.width)
let textureHeight = Int(frame.height)
guard let commandBuffer = commandQueue?.makeCommandBuffer(), let commandEncoder = commandBuffer.makeBlitCommandEncoder() else {
return nil
}
commandEncoder.copy(from: sharedVertexBuffer, sourceOffset: 0, to: vertexBuffer, destinationOffset: 0, size: vertexBufferSize)
commandEncoder.copy(from: sharedUVBuffer, sourceOffset: 0, to: uvBuffer, destinationOffset: 0, size: uvBufferSize)
commandEncoder.endEncoding()
commandBuffer.addCompletedHandler { _ in
let textureDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .rgba8Unorm, width: textureWidth, height: textureHeight, mipmapped: false)
textureDescriptor.cpuCacheMode = .writeCombined
textureDescriptor.usage = .shaderRead
self.texture = metalDevice.makeTexture(descriptor: textureDescriptor)
}
commandBuffer.commit()
Buffers are created by the Metal Device by specifying a length in bytes, and can be initialized with existing data such as the vertex and UV data above. Since this vertex and UV data will never change, access to that memory will be faster if it is transferred to the GPU to have private access exclusive to the GPU. i.e. The CPU does not need to change or do anything with this data after it creates it, so transferring it to the GPU optimizes the render pass since this data won’t have to constantly be copied from the CPU to the GPU every frame. To copy the vertex data to the GPU, I set up a command encoder and issue commands to copy the data from one buffer to the other that was initialized with storageModePrivate access. This is done for both the vertex and UV buffers. By adding a completion handler to the command buffer, I can be notified when this process is done, and then proceed to set up the texture object that will be passed to the fragment shader.
The Metal texture is created from a texture descriptor, which has been created with the width and height and pixel format of the texture. Some additional properties are configured for optimization purposes. The writeCombined option tells Metal that this texture will only be written to by the CPU, while the shaderRead option indicates that the fragment shader will only ever read from the texture. This texture will eventually contain the rendered bitmap from the game that will be displayed on screen.
Now let’s see how this is all put together in the renderFrame method of the MetalView class:
public func renderFrame(platformTexture: PlatformTexture) {
guard let metalLayer = layer as? CAMetalLayer, let drawable = metalLayer.nextDrawable() else {
return
}
semaphore.wait()
renderPassDescriptor.colorAttachments[0].texture = drawable.texture;
if let tex = texture, let textureBytes = platformTexture.texture {
let region = MTLRegionMake2D(0, 0, tex.width, tex.height)
tex.replace(region: region, mipmapLevel: 0, withBytes: textureBytes, bytesPerRow: platformTexture.bytesPerRow)
}
guard let commandBuffer = commandQueue?.makeCommandBuffer(),
let commandEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor),
let renderPipeline = renderPipeline else {
semaphore.signal()
return
}
commandEncoder.setRenderPipelineState(renderPipeline)
commandEncoder.setVertexBuffer(vertexBuffer, offset: 0, index: 0)
commandEncoder.setVertexBuffer(uvBuffer, offset: 0, index: 1)
commandEncoder.setFragmentTexture(texture, index: 0)
commandEncoder.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 6)
commandEncoder.endEncoding()
commandBuffer.addCompletedHandler { _ in
self.semaphore.signal()
}
commandBuffer.present(drawable)
commandBuffer.commit()
}
In order to draw to the Metal layer of the MetalView, we need a drawable from it. The texture of that drawable is then assigned to the color attachment of the render pass descriptor, which contains the target attachments for the render pass. This effectively says “store the result of this render pass into the drawable texture of the view’s Metal layer”. Next, I copy the texture bytes of the PlatformTexture from the game into the Metal texture object that will be passed to the fragment shader. Following that, a set of render commands are issued to the GPU: set the current pipeline state (containing the vertex and fragment functions to run), assign the vertex and UV buffers for the vertex stage, assign the texture for the fragment stage, draw the unit quad primitive as two triangles, and then finalize and commit the commands. The semaphore is used to ensure the render pass has completed before beginning a new one.
Finally, Metal shader functions go into a file with a .metal extension and are compiled as part of the build process. The shaders I use for my implementation are very straightforward:
using namespace metal;
struct VertexOut {
float4 pos [[ position ]];
float2 uv;
};
vertex VertexOut basic_vertex(constant packed_float3 *vertexArray [[ buffer(0) ]],
constant packed_float2 *uvData [[ buffer(1) ]],
ushort vid [[ vertex_id ]])
{
VertexOut out;
out.pos = float4(vertexArray[vid], 1.f);
out.uv = uvData[vid];
return out;
}
fragment float4 texture_fragment(VertexOut f [[ stage_in ]],
texture2d<float> texture [[ texture(0) ]])
{
constexpr sampler s(mag_filter::nearest, min_filter::nearest);
float4 sample = texture.sample(s, f.uv);
return sample;
}
The vertex shader simply assigns the incoming vertex and UV coordinates to the VertexOut type for each vertex in the draw call. The fragment shader does the simples texture mapping, using nearest filtering since the size of the unit quad primitive and the texture are exactly the same (i.e. the mapping from the texture to the quad primitive is 1-1).
That concludes this part, covering the graphics implementation of my custom game engine on iOS. For more information on writing a good software renderer, check out early episodes of Handmade Hero — a great series that inspired and informed me in the making of my game.
In the next part, I will be covering the other critical piece in any game: audio. For now, here is a short demo of a bunch of triangles (running at a lovely 60fps!):
flyingSand 