TILTBLEND: Tiltbrush And Blender


Hi everyone, I’m sorry for the radio silence – I hope that you and your loved ones are safe and well ❤.

Blender 2.8

For the longest time I’ve been postponing learning Blender. I tried to learn basic features in previous versions and always seemed to end up stumped by a labrythine UI much like the following screenshot.

ui nightmare

Blender is a specialized tool, and just like emacs or vim – the learning curve to power-user is intimidating. I had heard from friends that version 2.8 received a workflow and UI overhaul so I wanted to try learn it again. Gosh they were right! Not only is the UI intuitive, but the workflow is fantastic and easy to learn through experimentation. [If you have been afraid to try blender due to similar fear I would recommend you give version 2.8 a try. If you like it, and can afford it, please sponsor the foundation!]

Tiltbrush

Tiltbrush is also a specialized digital art tool – it is one of the most intuitive digital art tools I have ever had the joy to use. The only downside is it requires VR, and so it is quite isolating and can be somewhat inaccessible to folks with children. It comes with a set of beautiful brush shaders, some audio-reactive, so creators use it to make beautful reactive and very organic feeling creations.

tilt piece

My Learning Goals

I set out to learn Blender, and at the same time allow a user to create tiltbrush art outside of VR. I had a selfish motive for this, as I would like to use the gorgeous tiltbrush shaders to embellish some of my digital paintings. I came up with the idea of combining the two tools, and sketced out the idea. I named the resulting tool TiltBlend.

Building TiltBlend

Before making the tool I needed to understand the Tiltbrush file format and the Blender addon API. I will walk through the Tiltbrush format as I think it is quite interesting. The Blender API is well documented, and a pleasure to work with. For this reason, and for the sake of brevity, I will not include a description of that part of the process in this article.

Tiltbrush File Format

The Tiltbrush file is an custom archive compressed with zip with a prepended magic header which describes the file and makes it possible for Tiltbrush to identify its save files. The following files are inside of the uncompressed archive.

  • metadata.json
  • data.sketch
  • thumbnail.png

Tiltbrush Zip File Magic Header

The magic header is a simple structure, with the following format.

struct TiltHeader {
  unsigned int sentinel;         // uint32 ('tilT')
  unsigned short header_size;    // uint16 (currently 16)
  unsigned short header_version; // uint16 (currently 1)
  unsigned int reserved;         // uint32
  unsigned int reserved;         // uint32
}

Tiltbrush Metadata File

The metadata file holds information about the tiltbrush sketch, it is used to initialize and set up the Tiltbrush scene while loading it. It is also used as a lookup reference to apply brush materials to brush strokes.

{
  // This is a guid which maps to a couple of different environments that can
  // be used inside of tiltbrush.
  "EnvironmentPreset": "environment guid",

  // This is used to position the camera that is used to take the screenshot
  // which is used as the thumbnail for the tilt file. Its a standard transform
  // consisting of [[Vector3 position],[Quaternion rotation], Scale].
  "ThumbnailCameraTransformInRoomSpace": [
    [
      x,
      y,
      z                          // position
    ],
    [
      x,
      y,
      z,
      e                          // rotation
    ],
    s                            // scale
  ],

  // The entire scene has a transform, and this setting controls its position
  // inside the physical room. As above, this is a standard transform
  // consisting of [[Vector3 position],[Quaternion rotation], Scale].
  "SceneTransformInRoomSpace": [
    [
      x,
      y,
      z                           // position
    ],
    [
      x,
      y,
      z,
      e                           // rotation
    ],
    s                             // scale
  ],

  // This is one of the most critical lists in the file. It contains a list of 
  // guids which map to brush materials. Each stroke in the tiltbrush painting 
  // contains an integer which is a lookup index into this array. Tiltbrush uses
  // this mapping to apply brush materials to brush strokes.
  "BrushIndex": [
    "bush-guid"
  ],
  // This is the version of the metadata schema.
  "SchemaVersion": 2,

  // This section is optional, it is used to specify details of a model to be 
  // embedded into tiltbrush scene. It consists of a path to the model, a boolean
  // indicator of whether the model transform is constrained (pinned), a transform
  // for the model (with scale). It also contains a 'GroupID', I am not sure what 
  // this is used for yet.
  "ModelIndex": [
    {
      "FilePath": "test/cube.obj", // path to a model to embed
      "PinStates": [
        false                      // it the model pinned?
      ],
      "RawTransforms": [           // model 3d transform
        [
          [                        // vector3 position
            x,
            y,
            z
          ],
          [                        // quaternion rotation
            x,
            y,
            z,
            e
          ],
          s                        // scale
        ]
      ],
      "GroupIds": [
        0
      ]
    }
  ],

  // A scene can have a mirror which is used to create mirrored effects / symmetry 
  // in the scene. This section describes there this mirror is located in the scene. 
  // Again this is a standard transform consisting of [[Vector3 position], 
  // [Quaternion rotation], Scale].
  "Mirror": {                      // mirror tool transform
    "Transform": [
      [                            // vector3 position
        x,
        y,
        z
      ],
      [                            // quaternion rotation
        x,
        y,
        z,
        e
      ],
      s                            // scale
    ]
  },

  // I am not sure what this array controls yet. 
  "Videos": [],
  // I am not sure what this array controls yet.
  "CameraPaths": []
}

Tiltbrush Sketch Data File

To me, this was the most interesting sub-component of this file type to examine. As it is a custom binary format.

How a brush stroke is recorded

When the Tiltbrush artist holds a trigger on a controller and moves the controller through the air – a series of points, called control points, are recorded at a fixed rate or interval. The effect of this can be seen in the following plot of 2 brush strokes. The control points in this plot (in red) were extracted from a reverse engineered tiltbrush file (inset).

tilt piece

Each control-point consists of a position, a quaternion rotation, and then an array of int or float32 property values. This array is context based, and the context is determined by a bitmask set for each brush-stroke – this means that the array may contain different values and even different value formats.


struct ControlPoint {
  float x;          // 3d position, in float 32
  float y;
  float z;
  float qx;         // 3d rotation as quaternion, in float 32
  float qy;
  float qz;
  float qe; 
  // array of int/float32 depending on bits that are
  // set in a bit mask called controlpoint_extension_mask
  float[] or int[]
}

So a tiltbrush brushstroke consists of a brush index (which is an index into the BrushIndex array inside the metadata.json file), an RGBA color, a size value that represents the width of the stroke and a set of bitmasks. The bitmasks represent properties for both the stroke and the control points – they dictate how each is parsed.

struct Stroke {
  int brush_index; // this is an index into the brush array in metadata.json
  float r
  float g
  float b
  float a // brush color
  float brush_size;
  int stroke_extension_mask;
  int controlpoint_extension_mask;
  int32/float32 properties[] // for each set bit in stroke_extension_mask &  ffff
  uchar extensions[size]     // for each set bit in stroke_extension_mask & ~ffff
  int number_of_control_points;
  ControlPoint controlpoints[number_of_control_points];
}

When the stroke is rendered, a mesh is mapped along the control points. Then the material is applied. The material is retrieved by looking up the asset guid contained in the metadata.json BrushIndex array. The entire sketch is just a series of strokes formatted in the aforementioned way – and packed into the data.sketch file.

I was trying to concieve of a way to describe this with images. So I took the very first Tiltbrush drawing that I created which is pictured below.

tilt dragon

I then extracted all of the control point information out of this sketch – and I plotted it into a Unity3d scene. The form of the dragon above can be seen in the collection of control points.

tilt control points

Thats pretty much it! The extension masks described above control the following:

Stroke Extensions

  • flags
  • scale
  • group
  • seed

Control Point Extensions

  • brush pressure
  • timestamp

Tiltbrush Thumbnail

The last part of the tilt file is a thumbnail – this is just a PNG image. I used some Python magic to generate a PNG at runtime. This file is just black for now, but I’d like to improve this in the future and embed the filename of the generated Tilt file as image text.

Tiltblend

So this article is getting pretty wordy, I’ll try wrap it up. I took the above information, and I wrote an addon for Blender >= 2.8.

tilt blend ui

This addon currently allows a user to select vertices in the Blender scene and it converts these into tiltbrush strokes – it also allows the user to select brush properties. When the user is done, they can print out a Tilt file and then load it into Tiltbrush.

Voila!

tilt blend logo

The tool can be downloaded from my itchio. I hope that you have fun creating things. Feel free to add features.

https://glitch-land.itch.io/tiltblend