Premise

In late 2022, NVIDIA launched the NVIDIA Omniverse ecosystem to the public, which promised to be a very interesting platform. But in 2021, during their Beta phase, they announced some competitions to help promote the platform and let users explore the power of OpenUSD and NVIDIA’s rendering capabilities.

These competitions were called “Create With Marbles” because they provided a great deal of assets that they had been using in their Marbles demos, readily prepared for Omniverse users to be rendered beautiful within their ecosystem.

This was the second of their contests in that same year. where the first one involved doing single image renders, this one required a 1 minute video which involved physics to also show-off the new PhysX package included in Omniverse.

Goal

For this competition, the goal was clear. How to best show-off PhysX. I was very tempted to build some sort of Rube Golberg machine, but in the end I wanted to try and do something combined with another passion of mine. Music.

The competition title reminded me of an old video (and song) from Wintergatan called: Marble Machine. Where the musician builds an actual machine that plays a song by cranking it manually and letting physics take care of the rest. Marbles falling on a Xylophone, or Cymbals, even a base guitar! It’s incredible and worth watching it —> Wintergatan - Marble Machine

So I decided to build something of the sort with a version of this song within Omniverse.

Setup

Similarly to the previous competition, I didn’t document the systems and are now buried in some old hard-drives I need to recover.

Wintergatan provide the rights to the song for this kind of use, which made it easy to decide that I was going to re-create their song. They even provide the sheet music for it! I wanted to make my own music from it, but I realize I was time-poor so I found a MIDI version which I then modified to use for my purpose.

Then came the magic. I had a vague idea of how I wanted to show the different instruments to play. So I imported the assets into Houdini, and separating the instruments, then using the Midi file’s data, math and a little procedural magic, I was able to put together the basic concept.

Instrument 1 - The Xylophone

This consisted of a barrel that would get fed a bunch of marbles very randomly, but I expected that physics would easily distribute them evenly-ish. At least enough for a one minute simulation. What I did is figure out the notes that were needed for this part of the song, and passed that to Houdini’s system that would then place markers in the barrel to instance “chalk sticks” who’s only purpose was to pick up marbles so that the drop-off would happen at a very specific point where gravity would make them fall to some paint pots at the right time. From a mathematical point of view, it was easy enough to figure out considering the barrel is round and this segment loops constantly.

Instrument 2 - Drum

Probably the easiest of the 3. The drum is a consistent beat that happened a determined number of times. I didn’t even need the Midi file for this, I just counted the number of beats in a loop and used that to make a Ferry’s Wheel style contraption that would drop the marbles on some paint cans at a specific angle to hit the beat consistently. These were done in an angle so that they would retract and hit again on every revolution.

Instrument 1 - Guitar

This was interesting. In order to keep all the system contained, I wanted to see how much I could re-use. Since this relied on a repeatable loop. The xylophone barrel and the drum’s ferry’s wheel rotated together. That was the only “non-physics” rotation. It powered the whole system by rotating at a specific speed. So, how could I re-use this for this part? I extended the barrel on the sides to include 3 different rotating “trays” that would collect marbles at specific times, so that when these marbles reached the top they would fall onto slides that would make them hit a “marker pen” (guitar string?) at a the required time. This worked out pretty well!

Result

The result was the video you can see right here! I was very happy with how it turned out and loved all the moving pieces that made it happen. Given more time, I would have tweaked it further because Physics can be unreliable and every time I re-rendered I would get different results sometimes so getting everything to be timed perfectly was borderline impossible with the time I had.

However, it seems the judges were impressed too and I was incredibly proud to have won the competition! I was awarded an RTX A6000 card (signed by Jensen himself!) that I still use today for all my professional and personal development work.

Ever since then I still use Omniverse for various experiments or rendering tests. Professionally haven’t had many opportunities, but I still like to stay up to date with it because it has a lot to offer that other rendering engines can’t. It’s flexible, fast and the output speaks for itself!