This is a task that is currently in development right now. But is pretty far ahead by now.

When I joined the studio, changing the way a particular texture looked was heavily reliant on Decals, which was fine for a few decals but became hard to manage the moment we started populating loads of them and in particular, overlapping each other. What also became hard to address was keeping these decals without shifting when objects were completely dynamic (especially Skeletal Meshes).

So I decided to try something: “What if we could simply change the texture by painting on a texture itself?”- This carries it’s own risks too, but they can be overcome with careful consideration and would be far more performant than hundreds of decals. Especially when we start needing complete freedom to draw.

In this video you see the current state of the tool I developed (which combines some c++ logic, blueprints and shader work).

In this example, I’m demonstrating the difference of 2 “brushes” (we call them Inscriptors, but I think brush is clearer in this explanation). A Black Marker brush (a simple black linear drawing) and a Blood Drop brush, which also applies randomness to it’s splatter, and was attached to a marker for the sake of this demo:

Material Painter

This system is very extensible and gives the artists a lot of freedom to create their own brushes, but they have to be creative in how they do so because the system relies in a few strict considerations for this to work.

I’m going to outline the system which consists of two components: The Painter Actor and The Paint Layers Component (we name them differently as per our standards, but this will suffice for this demo).

Paint Layers Component

For this system to work, we rely on storing newly generated texture (render targets) which represents a Mask for each of the brushes we paint. In addition for a texture for each separate PBR attribute we are affecting (Diffuse, Normal, ORM, Emissive). In order to keep track of these newly generated textures, and make sure we’re updating the appropriate ones, upon attempting to paint a mesh, this component spawns, creates all the necessary textures (which get combined in the shader that is already prepared in case this gets activated) and stores them for any use.

For efficiency sake, it only creates the ones we are painting (not all potential ones), if a new brush comes into play, then these get added to the array.

Finally there’s a separate Master (to give it a name) set of textures, which return the final output of all the different textures combined (more on this later).

Since these textures get applied to the shader from the get go, any modification shows up and stays for good, unless they get erased (more on this later too). And we don’t have to worry about mesh location or the extra cost of overdrawing multiple decals as this action only gets performed once (during the painting process).

Paint Actor

This is where the magic happens. A lot goes on in terms of preparing the system, sanity checks, etc. But for the sake of this demo, I will explain how this whole thing works. And this will be explained in two parts, the drawing and the recording of data.

We’ll start with the recording of data because it will make the drawing examples below easier to understand. Essentially, if we want to draw a texture we have two obvious ways to go about it. We can create a texture by iterating through pixels, and applying whatever values we want to them, but this would be a CPU heavy system that would not be very performant, we want to use the GPU. And the easiest way to use it, is by using a Render Target capture. We have a separate camera, which only cares about a specific plane that is facing it and whatever the camera sees at a given time gets a snapshot and saves to a texture. That’s pretty much it. So the question is, how do we get it to show the correct thing?

Well, the first thought is always “What if I just get the UV position and draw directly there?”, that wouldn’t work because UVs are rarely consistent. So I needed a system that could work in World Space and somehow end in the UV Space.

What we do is the following, when painting, we change the shader to a specific one that the system knows about for the specific brush’s mask. For example, the one for the black marker looks like this:

MaterialMixer was how I originally named this, so these function names remain.

Draw Location Start, Control and End is a new feature I added because I want to be able to draw smooth curved lines which rely on Bezier calculations but is currently not fully implemented (hence missing from the video). When a paint action happens, the world space location of the paint is sent to this function and currently goes to all 3 inputs which then go to the BezierCapsuleMask function.

Draw Scale is the radius. Draw Alpha is the opacity.

What I want to focus on first is the World Position Offset - That’s what’s important for the Render capture. This function unwraps all the vertices to their relative UV space and offset to the location of the Render Target Capture:

With the vertices in offset to the Render Target region, the system all it has to do is take the snapshot and do whatever we decide to do with it after.

The rest of the function drives what the texture will look like. Let’s look at the Bezier Capsule Mask which requires a Position (the key factor here is that we are using the World Position EXCLUDING offsets, otherwise we’d get the UV Space ones). A Normal direction and the ABC points that make for a Bezier Curve calculation.

Inside the function, we essentially create a mask based on Bezier calculations (some SDF computation) in the form of a Custom Node HLSL script (a copy of the code can be seen here). Then further math for cases where we want sharp edges, smooth borders, etc… And we end up with what the stroke should look like.

Following the main shader. The MaskPass function converts that mask into a 3 colour channel that determines which of the PBR layers get affected if the shader includes them (R -> Diffuse + Emissive, G -> Normal, B -> Roughness).

And finally Operation is what determines the final output of the later. Are we adding to an existing layer based on a value? Are we overriding the mask value and setting the value instead? Or are we subtracting? This is what makes it powerful because we can have values that change over time, or simply erase them with other brushes (an eraser?).

We repeat this with all other PBR layers. Here we see Diffuse, setting to black, Emissive, also black, and ORM (Removing any metallic the asset would have) and Normals get skipped altogether, maintaining the Normals the asset had originally:

These get all snapshotted by the Render Target capture and saved the relevant PaintLayerComponent for the actor.

Now, how does an artist create a new brush? PaintActor has a dictionary where we define all new brushes, and the relative shaders that need to be considered when painting. And as an example to see how they can be different, this is the Mask layer for the Blood Drop:

Almost the same as the Marker Pen, except we are adding some noise to the World Position to give it the random splatter effect.

So, once all textures have been updated in every stroke, we reapply the original material and boom the texture is applied. The painting action itself clearly has the biggest performance hit, but it’s actually pretty fast and we don’t notice the impact in VR. Though I’d like to optimize things further and re-writing some of the existing blueprint logic in code. And if we move to Unreal 5, I’m certain I could make this more efficient by using Niagara 2D grids and Stages instead of Render Targets.

So, when I say that the artist needs to be creative in how the shader is applied, is that it must understand the limitations within the shader update (world position, uv, etc). But there are loads of possibilities! We can use this to draw cuts, make things appear wet, etc.

There’s a few extra details that go on here, like the mixing with the asset shader, and how the masks get used, etc. But I feel this document is large enough already and I feel this gives a pretty good understanding of what is going on.