td-rs
TouchDesigner is a lovely visual programming environment that allows artists to create live graphics workflows for video work, installations, and other creative projects. Users connect various types of nodes or "operators" together in a graphical network editor to build their project. TouchDesigner is also highly extensible, offering a C++ plugin API and Python scripting throughout the application. Writing plugins in C++ allows users to extend TouchDesigner's functionality by interfacing with external hardware or software APIs.
Unfortunately, writing C++ is kind of a chore. And all things being equal, we'd like to be able to
write our plugins in a crab based programming language that provides a more friendly modern workflow.
td-rs's goal is to provide a Rust plugin framework for TouchDesigner that allows creating
plugins with zero exposure to the C++ API.
As such, we have the following goals:
- 📦 A cross platform build system using Cargo
- 🦀 Idiomatic Rust plugin system
- 👷♀️ Type safe APIs where possible
- 🤔 Safety?? Maybe?
an aside comment on art and programming language theory
C++ might subject you to a constant series of paper cuts, but it's also a pretty good language for doing creative coding. Rust was originally developed to power #big #industrial code bases like a web browser, and its guaruntees around safety are intended to prevent the kinds of security issues that have plauged large C/C++ projects.
However, art code tends to not care about many of these benefits. Iteration speed is the reason to use a programming environment like TouchDesigner in the first place, and it's fine for projects to be a little broken or explode-y during the development phase. If you are doing something like accepting arbitrary user input in a way that could lead to a RCE, that sounds like a really cool project, but is pretty unusual. Usually most creative code is more like doing math on a single thread, not the kind of thing that requires an extreme commitment to safety.
Still, by fusing Rust with C++, we can have the best of both worlds. A more modern programming language for defining plugins that can still blow up in even more horrifying ways through abundant use of unsafe sketchy techniques outlined below!
Interfacing with C is already pretty tedious due to the need to uphold all of Rust's invariants when calling into foreign code, but C++ requires even more work to map things into C compatible types on both sides of the FFI boundary.
Thankfully, some of the brightest minds in the Rust community have created some truly magical solutions to help automate grafting these two languages together... ✨
cxx to autocxx
Initial experiments producing bindings to the TouchDesigner API used cxx, which aims to generate safe bindings to C++ APIs by requiring users to painfully describe the API surface in a series of limited types that are safe to pass to C. It the codegens a "bridge" which is used to pass these values between Rust and C++ and allow more complex things like calling foreign functions, receiving smart pointers and datastructures from C++ std, etc.
While this is magical, it also requires a lot of work. In the best case, Rust types in the bridge are
sufficient for interop, but due to the size and sometimes crufty nature of the TouchDesigner API, this
also required creating adapter code on the C++ side as well to facilite cxx's codegen, for example
avoiding things like raw pointers. Consequently, porting any given TouchDesigner functionality required
handwriting a multiple layers of code to present an idiomatic API in Rust: C++ adapter glue, the cxx bridge,
and higher level Rust interfaces.
Fortunately, there's an even more wild solution, autocxx
which attempts to generate the bridge for cxx to generate FFI glue. Meta-generation 🤯😵💫. Unlike cxx,
autocxx will attempt to generate bindings for an entire API simply by reading its header, and most importantly
to this project even includes the ability to magically generate a sub-class to a C++ abstract class, like
the ones defined for TouchDesigner plugins! Wow.
Switching to autocxx allowed completely automating generating bindings to the entire TouchDesigner API,
saving hours and hours of time and ensuring safety bridge code, which freed up time to do even more
dangerous unsafe things elsewhere.
plugin architecture
A TouchDesigner plugin is a DLL that exposes a subclass instance of a few different operator types. A series of lifecycle methods are called on this class in order to do work.
We define our RUst plugins as a simple Rust struct that implements a variety of required traits, that is then packaged in a library wrapper that is exposed to C++ via FFI, where it is then wrapped in a C++ sub-class that TouchDesigner actually calls. 🥵
Thus, when TouchDesigner calls a method on our plugin "class", the following happens:
- ~ TouchDesigner calls
execute - ~ C++ wrapper class calls inner extern C FFI function
- ~ FFI function calls interal
autocxxtrait representing our "subclass" on the Rust side. - ~ Internal trait calls the pugin defined operator trait that represents the actual custom operator functionality defined by the user.
At each layer, a variety of mapping and automating happens before handing over to the user's
actual custom code. The goal is for users to be able to create a plugin on simple Rust that implements
all the required traits just by calling a macro like chop_plugin! that hides all the layers of the plugin
onion from them. Ideally, a plugin can be defined using provided idiomatic traits without ever being
exposed to underlying FFI types.
build system
TouchDesigner relies on MVSC and Xcode to build plugins on Windows and macOS respectively. After spending so much time working with Rust and Cargo, trying to use these proprietary IDEs to build applciations is horrible and painful, exposing you to some of the worst yuck of cross platform native toolchains.
Our plugin is built by generating a variety of static libs built by Cargo (which also handles running
all the autocxx magic in build.rs). For a given library, we produce three libs:
- ~ A base lib that defines common operator functionality.
- ~ A operator type specific lib that defines functionality specific to the operator.
- ~ The actual plugin which implements all the required interfaces defined in the previous two libs.
These are then compiled using the platform toolchain which links these libs with our thin wrapper C++ class.
By using cargo-xtask, a custom task runner for Cargo, we attempt to provide a generic build system
that abstracts over the platform specific toolchains. Ideally, users should never have to know
about how the plugin is built other than invoking a single Cargo command that both compiles all
the Rust code and also produces the final plugin artifact.
doing unsafe things with proc macros
We use proc macros to provide two pieces of key functionality: the ability to derive and map operator parameters automatically and the ability to enable python enhanced functionality for a given operator.
mapping params and #[derive(Params)]
In the C++ API, mamaging params requires a lot of manual setup. For example:
void CustomSOP::setupParameters(OP_ParameterManager* manager, void* reserved) {
{
OP_NumericParameter np;
np.name = "Scale";
np.label = "Scale";
np.defaultValues[0] = 1.0;
np.minSliders[0] = -10.0;
np.maxSliders[0] = 10.0;
OP_ParAppendResult res = manager->appendFloat(np);
assert(res == OP_ParAppendResult::Success);
}
}
void CustomSOP::execute(SOP_Output* output, const OP_Inputs* inputs, void* reserved) {
double scale = inputs->getParDouble("Scale");
// ...
}
However, Rust's trait system can enable us to write something that eliminates this boiler plate. Our desired API looks like:
#[derive(Params)]
struct CustomSopParams {
#[param(labe = "Scale", min = -10.0, max = 10.0, default = 1.0)]
scale: f32
}
struct CustomSop {
params: CustomSopParams,
}
impl Sop for CustomSop {
fn execute(&mut self, output: &mut SopOutput, _inputs: &OperatorInputs<SopInput>) {
println!("{}", self.params.scale); // <-- "params" has been magically updated here
}
}
We can achieve this by defining two traits:
/// Trait for defining operator parameters.
pub trait OperatorParams {
/// Register parameters with the parameter manager.
fn register(&mut self, parameter_manager: &mut ParameterManager);
/// Update parameters from operator input.
fn update(&mut self, inputs: &ParamInputs);
}
/// Trait for implementing parameter types.
pub trait Param {
/// Register parameter with the parameter manager.
fn register(&self, options: ParamOptions, parameter_manager: &mut ParameterManager);
/// Update parameter from operator input.
fn update(&mut self, name: &str, inputs: &ParamInputs);
}
These traits are very similar, but will be used in our proc macro codegen to serve two different purposes.
First, we'll implement OperatorParams for our entire params struct that includes all our param definitions.
Then, we'll call the methods on Param for each individual field within that struct, allowing us to delegate
to the param types specific logic for registration and updating:
let register_field_code = quote! {
{
let options = ParamOptions {
name: #field_name_upper.to_string(),
label: #label.to_string(),
page: #page.to_string(),
min: #min,
max: #max,
};
Param::register(&self.#field_name, options, parameter_manager);
}
};
register_code.push(register_field_code);
let update_field_code = quote! {
Param::update(&mut self.#field_name, &(#field_name_upper.to_string()), inputs);
};
update_code.push(update_field_code);
let gen = quote! {
impl #impl_generics OperatorParams for #struct_name #ty_generics #where_clause {
fn register(&mut self, parameter_manager: &mut ParameterManager) {
#register_code
}
fn update(&mut self, inputs: &ParamInputs) {
#(#update_code)*
}
}
};
Then, before calling our plugin's execute method, we can check if it has opted into params
and call update so that they are at their latest value without any user intervention.
This also allows us to define parameter type that don't cleanly map to Rust primitives, and
even allow users to define behavior for their own custom structs. For example, TouchDesigner
has paramter types representing both a "file" and a "folder", which we want to map to a
PathBuf in Rust, but cannot derive automatically because the underlying type could represent
either param types. Consequently, we can simply implement a new type wrapping PathBuf, which
allows plugin implementors to choose their behavior:
/// A parameter wrapping a `PathBuf` that will be registered as a file parameter.
#[derive(Default, Clone)]
pub struct FileParam(PathBuf);
impl Param for FileParam {
fn register(&self, options: ParamOptions, parameter_manager: &mut ParameterManager) {
let mut param: StringParameter = options.into();
param.default_value = self.to_string_lossy().to_string();
parameter_manager.append_file(param);
}
fn update(&mut self, name: &str, inputs: &ParamInputs) {
self.0 = PathBuf::from(inputs.get_string(name));
}
}
Yay!
fun with python and #[derive(PyOp)]
Operators can be enhanced with Python, allowing users to script setting parameter values or calling arbitrary methods from Python. However, in C++ this requires a bunch of boilerplate setting up the required information for the Python C API. We can do better by generating this for our users, allowing them to simply implement a derive trait that can mark certain fields as Python accessors:
#[derive(PyOp, Debug)]
pub struct PythonChop {
#[py(doc = "Get or Set the speed modulation.", auto_cook)]
speed: f32,
#[py(get, doc = "Get executed count.")]
execute_count: u32,
offset: f32,
}
Using slightly evil proc macro techniques that involve casting void pointers from C++ back into
our concrete autocxx wrapper types, we can transparently update and retrieve values from our
Rust struct in Python without requiring the user to implement any other code.
Methods require a bit more manual interacing with the Python C API, but we also provide a mechanism to automatically register methods as Python:
#[py_op_methods]
impl PythonChop {
fn reset_filter(&mut self) {
self.offset = 0.0;
}
#[py_meth]
pub unsafe fn reset(
&mut self,
_args: *mut *mut pyo3_ffi::PyObject,
_nargs: usize,
) -> *mut pyo3_ffi::PyObject {
self.reset_filter();
let none = pyo3_ffi::Py_None();
pyo3_ffi::Py_INCREF(none);
none
}
}
While proc macros are always a little bit fragile and don't provide the best error messages without a lot of work, these macros combined can eliminate a huge amount of repetitive, error prone code and provide a really nice API that makes implementing operators in Rust more joyful.
typestate patterns for safety
One of the most powerful techniques in is using typestate to
leverage Rust's type system to make invalid operations impossible. For example, in the C++ API, it's extremely
easy to put the SOP_VBOOutput class into an invalid state that will crash at runtime:
void CustomSOP::executeVBO(
SOP_VBOOutput* output,
const OP_Inputs*
inputs, void*
) {
// Specify what to allocate in the VBO
output->enableNormal();
output->enableTexCoord(1);
output->allocVBO(1, 1, VBOBufferMode::Static);
// Do stuff with normals and tex coords
// DANGER!
// This segfaults because `enableColor`
// wasn't called above
Color* colors = output->getColors();
}
Although in this trivial example, fixing the bug is as simple as adding the missing call to enableColor,
it's easy to imagine a more advanced state where certain features of the VBO are enabled or disabled based
on some dynamic state like a parameter mapping.
We can make this bug impossible in Rust by designing the following simple state machine in the type system:
┌─────────┐
│Unalloc │
└────┬────┘
┌────▼────┐
│Alloc │
└────┬────┘
┌────▼────┐
│Complete │
└─────────┘
In Rust, we can thus define the following class:
pub struct SopVboOutput<'execute, State> {
pub state: State,
output: Pin<&'execute mut cxx::SOP_VBOOutput>,
}
The type parameter State here represents which state the output is currently in. While this field can contain
data, i.e. describing the allocated features of the VBO, it also can just be an empty struct:
// The possible states for SopVboOutput:
pub struct Unalloc;
pub struct ColorEnabled;
pub struct NormalEnabled;
pub struct TexCoordEnabled;
pub struct Alloc<N, C, T> {
pub vertices: usize,
pub indices: usize,
pub buffer_mode: BufferMode,
_normal: std::marker::PhantomData<N>,
_color: std::marker::PhantomData<C>,
_tex_coords: std::marker::PhantomData<T>,
}
pub struct Complete;
Note that the Alloc state itself has a variety of states that represent which features of the VBO have
been enabled. For example, the type Alloc<(), NormalEnabled, ()> has only allocated space for normals
in the VBO, which will be important later for guarunteeing safety. To allocate, we then define a number
of methods on impl<'execute> SopVboOutput<'execute, Unalloc> which consume Self and transition us
to the approate instance of Alloc:
impl<'execute> SopVboOutput<'execute, Unalloc> {
pub fn alloc_all(
mut self,
vertices: usize,
indices: usize,
tex_coords: usize,
buffer_mode: BufferMode,
) -> SopVboOutput<'execute, Alloc<NormalEnabled, ColorEnabled, TexCoordEnabled>> {
self.enable_normal();
self.enable_color();
self.enable_tex_coords();
self.alloc_vbo(vertices, indices, buffer_mode);
// etc...
}
pub fn alloc_normals(
mut self,
vertices: usize,
indices: usize,
buffer_mode: BufferMode,
) -> SopVboOutput<'execute, Alloc<NormalEnabled, (), ()>> {
self.enable_normal();
self.alloc_vbo(vertices, indices, buffer_mode);
// etc...
}
// ... And so on repesenting all possible states
// of enabled features
}
Returning Alloc with respective type parameters allows us to constrain the implmenentation of various
methods that require certain features to have been enabled prior to calling allocVBO:
impl<'execute, C, T> SopVboOutput<'execute, Alloc<NormalEnabled, C, T>> {
pub fn normals(&mut self) -> &'execute mut [Vec3] {
let normals = self.output.as_mut().getNormals();
if normals.is_null() {
panic!("this should never happen!")
}
unsafe { std::slice::from_raw_parts_mut(normals as *mut Vec3, self.state.vertices) }
}
}
This ensures that you can only call the method to retreieve the normals data if it has been properly allocated
first, which means we won't ever accidentally derference a null pointer. Once the buffer has been filled,
the user can call update_complete which consumes the output and ensures no other operations can be called on it:
impl<'execute, N, C, T> SopVboOutput<'execute, Alloc<N, C, T>> {
pub fn update_complete(mut self) -> SopVboOutput<'execute, Complete> {
self.output.as_mut().updateComplete();
SopVboOutput {
state: Complete,
output: self.output,
}
}
}
While the implementation of the typestate pattern is relatively verbose, using it is fairly easy, since type inference hides most of the complexity of the type parameters. In this way, we've transformed a relatively error prone dangerous imperative API into something that is safe. Cool!
conclusion
This project still has more distance to go to ensure safety and idiomatic API design in Rust, but hopefully can provide examples of ways that Rust can improve existing plugin APIs are provide a more friendly fun way to interact with C++, particularly in the domain of creative coding!