↫

Reflections on creative tech in Rust on Bevy's 5th Bday 🎂

Two years ago, I got involved in Bevy development through Nannou, the Rust creative coding framework. In Nannou, we were increasingly finding that Wgpu upgrades were quite burdensome and were looking for a way to decrease the maintenance costs associated with staying up to date, as well as find a broader community. Much to my surprise, two years later, I'm still doing Wgpu upgrades, just as part of Bevy's wonderful group of rendering contributors!

I continue to think that Rust in general and Bevy in particular offers huge benefits for creative technology. In this post, I'd like to reflect on where we are right now and where I'd like us to go in the next year with respect to rendering and creative tech.

What is creative tech? And why a game engine?

As the name implies, creative technology is a broad field of using... technology... for creative purposes! This can be anything from making digital signage, to VJing a rave, to fine art installations in museums. While economically a much smaller field than gaming, creative tech is all around us. Every day driving back from work as Salesforce Tower comes over the horizon, I think about how Jim Campbell's installation is seen by millions of people as part of their daily life. Corporate art? Yes, most definitely. But creative technology is increasingly public art, for example the installation I recently worked on for a new academic computer science building, which opens the possibility of real material interactions with its viewers in a way that goes far the screen.

Historically, creative tech and new media art has been written in every possible language. However, given its frequent reliance on computer graphics, audio, and weird hardware peripherals, C++ has a prominent role in its history. And Rust, at first, may seem like an awkward fit: artists rarely have the kinds of security concerns that demand memory safety and iteration speed is a top priority. Most art installations are built once and then never touched again, where the goal is to create something beautiful, not the most robust piece of software possible.

But C++ also has major downsides. Beyond it's technical problems that are obvious to any Rust developer, its community is often not the most friendly to people, like most artists, who may come from non-CS backgrounds. While Rust has a reputation for being a difficult language to learn, it's community and ethos much better supports people venturing into the world of weird creative uses of technology for the first time. Bevy is no exception here, as one of the most positive, supportive, and open communities I've worked in.

As a platform, Bevy also has the possibility to be so much more than just a game engine. The ECS architecture provides a kind of flexibility and modularity that allows viewing the engine as a kind of ecosystem of cooperating systems, rather than a monolithic desktop application designed for the singular purpose of making games. I think it's no surprise that some of Bevy's earliest commercial users include CAD desktop apps and there are people in the community doing scientific simulations in Bevy. In many ways, more than an engine, Bevy is a kind of research platform for graphics and application development, and it's my hope in the future that it will be the best option for doing creative technology projects.

The Past Year

Prototypes

I've also done a number of prototypes exploring what's possible in the current Bevy API:

  • Integrating Bevy inside TouchDesigner using Wgpu and Vulkan/CUDA interop. TouchDesigner is a dominant engine and editor for installation work, but its renderer isn't always the most performant. Being able to embed Bevy in TouchDesigner allows import textures and other data from TouchDesigner into Bevy to take advantage of our new GPU driven renderer!
  • Running PyTorch and CUDA inside Bevy's render graph for real time computer vision tasks. Machine learning is an important part of modern creative tech applications, and I wanted to demonstrate the possibility of running CUDA workloads inside our render graph by exporting Bevy's textures into CUDA.
  • Compute based DMX pixel mapping. Working with LEDs is complicated and developing tools to better support that workflow is oen of the key goals of Nannou!

Nannou

Nannou's Bevy integration continues to mature and is getting very close to being ready to release. Beyond porting our application model to run on top of Bevy, I'm most excited about some of the following features that I implemented in the past year:

Bevy development

For most of the past year, I've focused mainly on contributing to the renderer development process itself rather than building out the ecosystem of tools required for creative technology. This work has included two major contributions which I'm proud of:

  • Cold specialization, which involved rewriting much of the CPU side queuing and specialization code in the renderer to support retained rendering and broader efforts by others to implement GPU driven rendering.
  • Type erased materials, which is an incomplete part of ongoing work both to bring our 2d and 3d renderers closer together, but also to open the possibility for material API exploration and integration with our ongoing editor efforts.

What's coming

For the next year, I have a number of goals in mind, both for Bevy's renderer and for supporting creative technology in general. I'm excited by a number of efforts by others, most notably the new audio working group, that will directly benefit creative use, as well as work on the editor that will hopefully form the basis for a future TouchDesigner-like application built entirely in Bevy.

Materials ❤️ WESL

I'm thrilled about our plans to transition to WESL and I plan to continue to focus on ways that we can improve the material API and support artist driven shader authoring in Bevy. This includes paying down a substantial amount of technical debt in our renderer but also experimenting with new APIs that allow for more dynamic, runtime workflows including hot-reloading, code generation and shader reflection in WESL, and an eventual shader graph in the Bevy editor. There's a lot of work to do here and a lot of research required to explore the possibilities for what an ECS first material API might look like.

Asset streaming

In collaboration with a number of other contributors, I really hope to make progress on improving support for more advanced techniques for managing VRAM, specifically streaming assets. This is a friction for broader commercial adoption of Bevy but has also been a continual issue in creative tech projects I've worked on where we often deal with huge assets and a variety of hacks in order to make sure they get where they need to be on time.

Release Nannou

Finally, I'm thrilled to be very close to finally releasing Nannou. It's been in a pretty good place for use with Bevy for several months now, but just needs one more little push to get over the line and have a general release. I'm hoping we can coordinate with the 0.18 release of Bevy. Beyond that, I'm excited to use Nannou as a hub for all the kinds of weird Bevy plugins that don't make sense upstream. Nannou already has great support for things like working with lasers, but I think we can push this further in terms of integrating with all other kinds of hardware SDKs that creative tech application developers expect: cameras, sensors, etc.

Vulkan / Cuda

I continue to take great interest in the way that we can integrate Bevy with Vulkan and Cuda via Wgpu interop. My plans here are to polish some of the existing work I have to enable running ML workload in Cuda via Pyo3 and additional to implement some creative tech specific features for multi-gpu rendering using Mosaic and Direct Display.

To the next year of Bevy!

Bevy is such an incredibel community, and I'm so thrilled to see what work we get done in the next year. If you're interested in participating in some of this work as it relates to creative tech in Bevy, please be in touch! ✨