Idea/Goals

For a recent performance, I wanted to be able to integrate a real time simulation with my Tidal performance. I considered having the simulation (which was running in real time on a separate machine, forwarding data to my machine via OSC over UDP) send messages directly into SuperCollider, where they would be parsed and used to modify parameters on various signal processing elements. But this seemed a little too “locked in”, i.e., not flexible and improvisatory.

Instead, I wanted to see if there was a way that I could connect the simulation data to the various parameters that are already available with TidalCycles–things like “#coarse“, “#speed“, “#gain“, “#pan“, etc. Ideally I wanted to have this be flexible enough that I could make and break these connections during the performance as I wished, bringing the simulation into the performance as an interactor that could be “plugged into” by any TidalCycles element.

The simulation that I was working with was a fluid simulation on a two dimensional plane, with “tracers” dropped into the simulation which could be queried for positions over time. With the simulation being a two dimensional space, it was an obvious move to try this idea out connecting tracer position to gain and pan in a multichannel setup. We set up a 4-channel system in a circle around the performance space.

SuperCollider Boot Code:

(
s.options.device="M-Track Eight"; //Multi Channel Audio Interface
s.options.numBuffers = 1024*16;
s.options.memSize = 8192 * 16;
s.options.maxNodes = 1024 * 32;
s.options.numOutputBusChannels = 4; //Number of Output Channels
s.options.numInputBusChannels = 2;

s.waitForBoot {
	~dirt = SuperDirt(4, s); //Set Number Output Channels here too!
	~dirt.loadSoundFiles("(SoundSamples_Folder)", false);//Manually loading samples folder.
	s.sync; //Must wait for server sync.
	thisProcess.openUDPPort(57124); //Open another UDP Port for Intercept Script
	~dirt.start(57124, [0, 0, 0, 0, 0, 0, 0 ,0 ,0]); //initialize SuperDirt on that listening port. Initialize SuperDirt with 9 orbits.
};
)

TidalCycles Intercept Script (python)

from pythonosc import dispatcher, osc_server, udp_client, osc_message_builder
import threading
import numpy as np

import pprint
pp = pprint.PrettyPrinter(indent=4)

#Send to SuperCollider using this client
client = udp_client.SimpleUDPClient("localhost", 57124)

#Send parameters to visualizer (built with openFrameworks) here
visClient = udp_client.SimpleUDPClient("localhost", 8881);

OrbitParams={}
normalizeTracerCoords=True

#These are all the possible attachments, along with default scaling values
validAttachments={"hcutoff": (1000, 5000),
                  "bandf": (500, 5000),
                  "bandq": (0.25, 5.0),
                  "begin": (0.0, 0.5),
                  "coarse": (0, 64),
                  "crush": (16, 2),
                  "cutoff": (500, 2500),
                  "delay": (0.0, 1.0),
                  "delayfeedback": (0.0, 1.0),
                  "delaytime": (0.0, 1.0),
                  "end": (0.5, 1.0),
                  "gain": (0.0, 1.0),
                  "accelerate": (-5.0, 5.0),
                  "hresonance": (0.0, 1.0),
                  "loop": (0.1, 4.0),
                  "pan": (0.0, 1.0),
                  "resonance": (0.0, 1.0),
                  "room": (0.0, 1.0),
                  "shape": (0.0, 1.0),
                  "size": (0.0, 1.0),
                  "speed": (-2.0, 2.0)}

attachmentDelimiter="_"

#Clamp to min-max.
def clamp(n, minn, maxn):
    if n < minn:
        return minn
    elif n > maxn:
        return maxn
    else:
        return n

#Scale data from Simulation to given range and clamp
def scaleAndLimit(val, outMin, outMax, inMin=0.0, inMax=1.0):
    if (inMin != 0.0 or inMax != 1.0): val = (val-inMin) / (inMax-inMin)
    if outMax0:
        if attachedParams[0] in validAttachments:
            if attachedParams[0] in args:
                eIdx=args.index(attachedParams[0])
                del args[eIdx+1]
                del args[eIdx]

            if len(attachedParams)>=2 and isFloat(attachedParams[1]):
                thisMin=float(attachedParams[1])
                if len(attachedParams)>=3 and isFloat(attachedParams[2]):
                    thisMax=float(attachedParams[2])
                    del attachedParams[2]
                else:
                    thisMax = validAttachments[attachedParams[0]][1]
                del attachedParams[1]
            else:
                thisMin = validAttachments[attachedParams[0]][0]
                thisMax = validAttachments[attachedParams[0]][1]
            try:
                if (attachTo==['v']):
                    thisVal = scaleAndLimit(OrbitParams[orbitNum][attachTo], thisMin, thisMax, inMin=0.0, inMax=0.01)
                else:
                    thisVal = scaleAndLimit(OrbitParams[orbitNum][attachTo], thisMin, thisMax)
            except KeyError as e:
                print("[KEY ERROR]\t", e)
                print("Orbit Params")
                print("Orbit Num {}".format(orbitNum))
                pp.pprint(OrbitParams[orbitNum])

            args.append(attachedParams[0])
            args.append(thisVal)

        del attachedParams[0]

#This is the function attached to listener from Tidal
def getFromTidal(unusedAddr, *args):
    args = list(args)
    global client
    if "orbit" in args:
        idx = args.index("orbit")
        orbitNum = int(args[idx+1])
        if(orbitNum in OrbitParams and orbitNum>0 and orbitNum<=8):
            #Each item here looks for attached arguments and if present, parses the relevant following arguments into new message parameters. 

            if "attach_X" in args:
                attachedX_idx = args.index("attach_X")
                attachedX_params = args[attachedX_idx+1].strip().split(attachmentDelimiter)
                del args[attachedX_idx+1]
                del args[attachedX_idx]
                parseAttachments(args, "x", attachedX_params, orbitNum)

            if "attach_Y" in args:
                attachedY_idx = args.index("attach_Y")
                attachedY_params = args[attachedY_idx+1].strip().split(attachmentDelimiter)
                del args[attachedY_idx+1]
                del args[attachedY_idx]
                parseAttachments(args, "y", attachedY_params, orbitNum)

            if "attach_R" in args:
                attachedR_idx = args.index("attach_R")
                attachedR_params = args[attachedR_idx+1].strip().split(attachmentDelimiter)
                del args[attachedR_idx+1]
                del args[attachedR_idx]
                parseAttachments(args, "r", attachedR_params, orbitNum)

            if "attach_T" in args:
                attachedT_idx = args.index("attach_T")
                attachedT_params = args[attachedT_idx+1].strip().split(attachmentDelimiter)
                del args[attachedT_idx+1]
                del args[attachedT_idx]
                parseAttachments(args, "t", attachedT_params, orbitNum)

            if "attach_V" in args:
                attachedV_idx = args.index("attach_V")
                attachedV_params = args[attachedV_idx+1].strip().split(attachmentDelimiter)
                del args[attachedV_idx+1]
                del args[attachedV_idx]
                parseAttachments(args, "v", attachedV_params, orbitNum)

            visClient.send_message("/hit", orbitNum-1)
    print(args)
    #'args' is now a modified version that starts with the performer's executed block from Tidal, augmented based on simulation data and requested attachments.
    client.send_message("/play2", tuple(args)) #forward new message to SuperDirt (SuperCollider)

TidalRecvDispatcher = dispatcher.Dispatcher()
TidalRecvDispatcher.map("/play2", getFromTidal)
TidalOutServer=osc_server.ThreadingOSCUDPServer(("127.0.0.1", 57120), TidalRecvDispatcher) #This is the OSC/UDP input from TidalCycles (GHCi). By default, TidalCycles tries to sent to port:57120 (the standard SuperCollider server port). We want to snag those messages here.

def car2pol(x, y):
    return np.sqrt(x**2+y**2), ((((np.arctan2(y,x)/np.pi)+1)/2)-0.25)%1.0

def prepOscData(x, y, limitToNormalCoords=False):
    r, t = car2pol((x*2)-1, (y*2)-1)
    if limitToNormalCoords:
        r = clamp(r, 0.0, 1.0)
        t = clamp(t, 0.0, 1.0)
    return r, t

import socket
sock=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(("10.0.1.13", 8000)) #This is my IP address and port for listening to messages sent from the simulation. 


def getVelocitySquared(vx, vy):
    return (vx**2) + (vy**2)

def getVelocity(vx, vy):
    return np.sqrt(getVelocitySquared(vx, vy))

#Function for dealing with data from Simulation. 
def udpRecv():
    global OrbitParams
    while True:
        data, addr=  sock.recvfrom(1024)
        if data is not None:
            d = data.decode("utf-8")
            dataSplit = d.split(" ")
            visOut = []
            dataSplit[:] = [x for x in dataSplit if x != '']
            for x in range(len(dataSplit)//2):
                thisX = float(dataSplit[x*2].strip())
                thisY = float(dataSplit[(x*2)+1].strip())
                r, t = prepOscData(thisX, thisY, normalizeTracerCoords)

                if (x+1) in OrbitParams:
                    OrbitParams[x+1]['r'] = r
                    OrbitParams[x+1]['t'] = t

                    OrbitParams[x+1]['vx'] = thisX - OrbitParams[x+1]['x']
                    OrbitParams[x+1]['vy'] = thisY - OrbitParams[x+1]['y']

                    OrbitParams[x+1]['avx'] = np.abs(OrbitParams[x+1]['vx'])
                    OrbitParams[x+1]['avy'] = np.abs(OrbitParams[x+1]['vy'])

                    v=getVelocity(OrbitParams[x+1]['vx'], OrbitParams[x+1]['vy'])
                    if (v < 0.02): OrbitParams[x+1]['v'] = v
                    OrbitParams[x+1]['x'] = thisX
                    OrbitParams[x+1]['y'] = thisY
                else:
                    thisOrbitParam = {'r': r,
                                      't': t,
                                      'x': thisX,
                                      'y': thisY,
                                      'vx': 0.0,
                                      'vy': 0.0,
                                      'avx': 0.0,
                                      'avy': 0.0,
                                      'v' : 0.0

                    }
                    OrbitParams[x+1] = thisOrbitParam

            for idx, tracer in OrbitParams.items():
                visOut.append(tracer['x'])
                visOut.append(tracer['y'])
                visOut.append(tracer['v'])

            visClient.send_message("/msg", visOut)
#Run UDP Receiver (from Simulation) on a separate thread.
UdpRecvThread = threading.Thread(target=udpRecv)
UdpRecvThread.start()

try:
    TidalOutServer.serve_forever()
except KeyboardInterrupt as e:
    print("quitting", end='')
    TidalOutServer.shutdown()
    sock.close()
    UdpRecvThread.join()
    print("\t\t[DONE]")

TidalCycles Startup

let (attach_X, _) = pS "attach_X" (Just "")
    (attach_Y, _) = pS "attach_Y" (Just "")
    (attach_R, _) = pS "attach_R" (Just "")
    (attach_T, _) = pS "attach_T" (Just "")
    (attach_V, _) = pS "attach_V" (Just "")
    tracer0 = (#orbit "1")
    tracer1 = (#orbit "2")
    tracer2 = (#orbit "3")
    tracer3 = (#orbit "4")
    tracer4 = (#orbit "5")
    tracer5 = (#orbit "6")
    tracer6 = (#orbit "7")
    tracer7 = (#orbit "8")

let followTracer = ((#attach_R "gain_1.0_0.75") . (#attach_T "pan_0.0_1.0"))
    spacey a=((#attach_X "delay_shape_0.1_0.5"). (#delayfeedback a) . (#attach_Y "gain_1.0_0.5_delaytime_pan"))

I've added the above definitions to my TidalCycles startup routine. This let's me use "attach_X" as I would any other parameter in Tidal (i.e., "#pan")

The messages here are a bit messy, but given the short turnaround time for this project and wanting to just get the thing working, this is where things are at for the moment:

You can call any of the "attach_*" items followed by any of the valid attachments as listed in the beginning of the Tidal Intercept python script. By default, this will use the defined min and max values as given in the python script to scale the simulation data. If you want to provide custom scaling parameters, follow the parameter name with two float values, for min and max, separated by an underscore ("_"). So... #attach_X "pan_gain_0.5_1.0" would set the pan and gain of this item based on the X position of a tracer in the simulation, scaling the pan to be from 0.0 to 1.0 (using the default values in the python script), and scaling the gain from 0.5 to 1.0.

The last step to getting this to work is to attach the item to an actual tracer or entity in the simulation. Since a lot of the signal processing in TidalCycles/SuperDirt is "orbit" specific, I just decided to use #orbit "" as the parameter to choose which simulation entity to attach to.

d1 $ tracer0 $ gain "1(5,16)" #n (irand 15) #s "house_kick" #attach_R "gain" #attach_T "pan"

The end effect, here, is that we can attach sounds, stacks, and any TidalCycles entity to data coming from some other realtime application to any of the TidalCycles synth parameters. This example, directly above, will attach the line to tracer0, modifying both gain and pan. Gain is based on R (radius, or distance from the center), and Pan is based on T (theta, or rotation around the center).

There are several major shortcomings here to this process. The biggest one is relying on Python's string search to match parameters. It's slow. Super slow. Needing to do lots of this string search rapidly can make things a little chunky temporally, causing some messiness by the time sound comes out of SuperCollider. There's definitely some improvements that could be made to help this.

Also the formatting of the attachment messages is horrendous right now. It's functional, but it ain't pretty.

The simulation data that the Intercept script is receiving could be ANY data that is normalized to 0.0 to 1.0. I can't share the specific simulation that I was using, here, because it was developed by Prof. Chris Rycroft (Harvard School of Engineering and Applied Sciences). But the formatting for this data is simply a long string, space-delimited, containing a series of floating point values. Each pair of values constitutes the X and Y coordinates of a single entity. So if I have 8 things to watch, the Intercept script should receive a string with 16 space-delimited floating point values.