nino project
modina residency

documentation 2025

background

shadows play on weathered textures, landscapes both natural and surreal, light dances on surfaces virtual and visceral. avatars breathe, embodying the earth, sky, and the storm within, drifting between worlds, animated by the unseen currents of a dancer’s breath, captured in the pulse of motion. here, the audience no longer watches—they become space where the lines between player, performer, and witness blur — focusing on movement’s essence over physical traits. ai images inspired by joan skinner’s ‘releasing dance technique,’ are modelled as 3d characters and animated using motion capture data. mocap systems allow viewers to inhabit avatars, alongside a dancer.

about:

artists gibson/martelli are supported by modina.eu during a residency at trafó, budapest with music from mechanoid muse: marcus villaca and dancers: balázs oláh, csilla nagy, mar serra martí and helén tamaskó together with eszter gál and special thanks to liza nagy, motionbank mark coniglio and with clothing from kitti laposa.

video of parts of the performance and some behind the scenes shots

audience animating avatars at trafó

comments, feedback etc. gibsonmartelli@gmail.com last update: 06/07/25

ai mocap and tracking systems

we want to use live markeless mocap, either for audience interaction, performers or both, the reason for markerless is ease of use, participants dont have to wear mocap suits. the first system we looked at is the captury and we have a great hands on demo day with target3D in their london studio. here is the video from that day:

the system uses either windows with 8x optitrack cameras or with linux and 8x FLIR cameras. the system can track 4 people if you have a good space. the issue with it is, it’s really expensive. the next most similar system is newcomer moveai -this uses a similar camera array setup, however we have not had the opportunity to try this.

finally we discover new kid on the block radical, this system uses a single camera to send video up to a web-based ai and sends back skeletal data. the web space can hold up to ten performers each dialling in on a separate pc or phone. The best thing, its really inexpensive. it had some issues with foot sliding & latency but hey what do you expect for 100 dollars a year. they are updating it frequently - and in fact on 07/03/25 fixed the footsliding issue with version 4 in livemode! radical did a case study of nino here.

heres a short video of testing radical:

in the video, the RADiCAL website getting the live video from the webcam is on the left, below is the RADiCAL avatar showing the mocap, on the right is the mocap data streamed into Unreal, animating 3 avatars. you can see there is a lag between person moving and the Unreal avatar moving. this is mainly to do with internet upload speeds (I think). However creates a quite nice feedback where you copy the avatars movement, -- which is your movement. the single camera copes well with a bad lighting setup too. over the course of the residency the developers continue work on the software implementing better control over foot placement to reduce sliding. (the other great feature of the system is that it fails gracefully - when the tracked subject leaves the volume, the avatar doesn’t do anything weird - instead freezing in a nice position.)

another challenger in the arena of ai mocap is remocapp --this system can use up to 8 cameras, supports face and fingers and has an unreal plugin, and is not expensive. We are testing this, one of the immediate issues is that the calibration process for each camera seems a little tricksy. we are testing with two cameras, the data is processed locally with internet connection only needed for licence checking. initially the skeleton, which seems smooth inside the remocapp software, appears to misbehave when streaming to unreal 5. after contacting the developers we learn to separate the 2 cameras to be at least 5m apart which gives a much better result.

heres a short vid of testing remocapp

on the left is the software, I have 2x webcams setup. on the right is the data streaming into unreal. the software uses a local AI model rather than streaming to the web and can support up to 8 cameras. I could not get a great result with it and I tried several times with different configs. the advice from developer is to keep cams at least 5m apart and at head height. there is a calibration marker for the cameras and a slightly tricksy calibration process. I had high hopes but eventually preferred RADiCALS ease of use and graceful failures.

Rehearsing with RADiCAL and dancers at SIN in Budapest - we’ve set up 2 devices captuting and streaming into Unreal eventually we get three at once.

another osc system we try is posenetosc - this is made with openpose. naoto from modina makes an .exe for this 2d opensource skeleton detection and tracking system. its quick , web-based and we make a widget in unreal to smooth incoming data and use it to ‘scratch’ a prerecorded animation sequence, controlled by a joint position from a user in moving of the camera, using the method below.

Open pose detecting 2 performers. This is quick and the segmentation (what limb is what) is good but of course its only 2D

ai camera

testing an ai camera with dancers, we got this 4k ptz camera from obsbot: initially because its possible to send it commands via osc - we considered making the camera behave like an automated follow spot and using mocap data as in input. It tured out that it has an AI feature that can track humans and it sort of behaves like a cameraperson when its in a larger space, it can follow movement or isolate the lower part of the body and gives pleasing results.

diary

audience interaction with avatars after the performance

artists talk at Trafó

Naoto from Modina makes a 3D scan of the Trafó main hall and we add it to Unreal to experiment with staging/lighting options, alongside making a physical model of the space.

Audience trying the markerless mocap system after the performance

nino teaser from Trafó

rehearsal shot

performance shot

testing ReMocapp wih Will and Naoto (all these images above from maÌteÌ kalicz

choreography

skinner releasing technique (srt) facilitates the release of tension in the body, enabling greater freedom of movement and challenging habitual movement patterns. it invites engagement with the imagination through poetic imagery, offered by the teacher in clusters. central principles include weight, buoyancy, and the belief in an innate, primal grace shared by all bodies.

for this project, we drew upon the ethos of srt—its energetic fields, philosophy, and guiding principles. we explored concepts such as body economy, the idea that “less is more,” and the quiet strength of gentleness. our focus was on translating key principles of srt into performance, embedding aspects of its pedagogy into both choreography and stage presence.

modina offers a framework capable of reaching deeper into societal layers, blending embodied knowledge with ai to interrogate how we inhabit the world. the stage becomes a magnifying lens for life itself. what truths or sensations might we unearth here that have remained inaccessible until now?

one central question we explored was: how does information move through the body?

the rehearsal process involved five dancers: one with extensive experience in srt, one with some prior exposure, and three with none. due to the tight timeframe, each rehearsal session began with an srt class, allowing the dancers to gradually absorb and embody the work.

following auditions, we identified an interest in seeing balázs engage with srt—particularly intrigued by how it might influence his movement, inviting sudden, supple moves.

upon encountering the basement performance space at trafo, i was inspired to incorporate whispering on stage—an element drawn from the srt pedagogy. this choice aimed to introduce an air of suspense and ambiguity at the beginning of the performance. i was also curious about how whispering, with its asmr-like qualities, could communicate kinaesthetic experiences and provoke a sensory response in the audience.

studio practice overview

compositional structures:

groupings: solos, duets, trios, full group work.
tasks: following and leading, shadowing, not touching, shifting energies, finding balance, and changing places (inspired by ai tracking camera).
movement qualities: slowing down, gliding, skimming, sudden shifts, subtle lifts, and phantom gestures.
spatial concepts: figure-of-eight pathways, entering/exiting around seated audience, tipping, falling, melting, and rolling.

srt-inspired elements:

key classes referenced:

class 3: rolling shoulders, counterbalance into rise
class 4: sighing, readiness, letting go
class 7: sudden supple moves
class 8–9: unexpected moves, whispering, tiny lift-off, gliding, “river running,” and skimming

key concepts:

whispering nonsense growing louder, traveling through the body
spongey squats, nudging pulse, instant readiness
luxuriating into space, phantom hands, laying out
whispering as asmr/kinaesthetic experience

ai & technology integration:

ai camera influences spatial choices: changing places, leadership dynamics, and ‘stealing the camera’ game.
dancers explore themes of ai replacing nature in virtual landscapes.
sound/sensory layering:

weather effects (visual/sound)
animal words (not sounds)
fast music with stillness, slow music with frenetic movement

improvisational & game-based tasks:

‘blind copy,’ ‘steal the camera,’ ‘imaginary partner’ folk improvisations.
port de bras as duet without physical connection.
onomatopoeia and amanuensis (sound-based imitation, following/leading).
duets include skipping, counterbalance, tipping with a partner.

choreographic themes & motifs:

recurrent imagery: nudging dance, hair/crown of strings, leaning back (software metaphor), dark weather passing.
embodied metaphors: dog nuzzle (constant connection), failing gracefully (radical), nature vs ai.

themes and energies

human / non-human energies:
exploring the interplay between human and non-human elements—how energy manifests across bodies, technologies, and environments.
radical / ai camera:
investigation into ai-based pose estimation and how ai influences choreography, perception, and presence on stage.
embodied focus:
special emphasis on hands, weight, buoyancy, and floating qualities in movement—linking physicality to thematic concepts of transformation and presence.

music

sound arc development:
testing a range of tracks from marcus’ albums to shape a cohesive sonic arc for the performance.
new compositions:
marcus is composing new pieces inspired by onomatopoeia and guitar chord resonance, using an a.i system suno.ai to generate tracks - text prompts describe the genre and mood of the song, lyrics, and melody.
performance format:
marcus will not perform live but will contribute via audience-interactive sound elements during the show.

visual design

midjourney visuals from srt prompts:
digital avatars and environments were developed based on srt-inspired prompts, including:

characters:

blob
ash
sphere
folk dance avatars

environments:

blob world
brueghel-style black and white scene
testbox
concrete bunker
trash world
forest
desert
succulent garden
sky with translucent spheres

technology & motion capture

pose estimation & camera setup:

tested 2 ai-based pose estimation systems.
set up front and side camera views for 3d tracking.
best results found on depth-axis tracking.

motion capture tools tested:

openpose (python-based):
3d support, best for single-performer capture, studio use only—no real-time mode.
theia3d / captury / move.ai / radical:
explored capabilities and limitations across multiple platforms.

technical infrastructure:

systems tested for:

ease of use
aesthetic integration with movement language
stability (crashability)
latency
real-time capability
prototype feasibility

tools and resources:

windows-based systems
poe switch for bandwidth/data
post net system
linux flir cameras
motion bank tracker license
sound/ml/webtech integration using mc and python
motion bank mac code
physical dongle for optitrack (with 1-week delivery lead)
captury demo and computer tracker licenses

dramaturgy / stage design
concept ideas and future possibilities

1. dual space installation – capture station landscape

a stage set featuring a sculptural "cut-out" landscape with two distinct yet interconnected zones:

space 1: live or pre-recorded performance by one of the dancer-characters.
space 2: live or pre-recorded music, inspired by the winterscape format used at event gallery.

integrated with a web interface, allowing interaction, observation, or feedback loops between audience and performers.

2. object manipulation & capture

a dancer engages in a live interaction with objects that are optionally captured or tracked, emphasizing the relationship between movement, matter, and perception.

3. architectural transformation

the theatre’s curved roof is treated as a dynamic scenographic element, transforming the spatial atmosphere—evoking scale, enclosure, or celestial themes.

4. dmx lighting & gesture-controlled color

use of dmx lighting system with gesture-responsive controls, linking dancer movement to live color and sound modulation—creating a synesthetic interplay between gesture and environment.

5. ai and the natural world – a novacene perspective

themed around the tension between ai and nature.

one thread explores ai as a threat or replacement for nature.
another, inspired by james lovelock’s novacene, suggests ai and humans co-evolve as planetary stewards—working together to preserve the earth.

6. immersive entry – blind copy installation

the audience enters an installation-like environment where dancers are already performing blind copy, creating intimate and proximate encounters.
emphasizes immediacy, perception, and mimicry.

7. chromatic shift – from monochrome to colour

a visual dramaturgy where the performance moves from a black-and-white aesthetic into a world of colour, signifying transformation, awakening, or technological/human evolution.

audio releases

mechanoid muse released two albums from the nino project on bandcamp, nino and onomatopeia

technical

set up characters in unreal for live mocap retargeting

livelink

play takes

there are a lot of steps:

1. first i make a super basic stickman character that’s the same size as ruth using the same skeleton as unreal’s default quinn. this is the skeleton that i will send mocap too. this character has an ik rig, and two animation blueprints, one that I can send mocap too, via a livelink pose node and a second that plays back some prerecorded ruth takes, including a rom, very useful for testing.

ik retargeter

2. for each custom character im going to make an ik rig, and an ik retargeter. inside the ik retargeter, i set the source to my ik_stickman and the target to the customcharacter ik rig --there is a bunch of fiddling that can be done here, to get a good retarget --what really helps is that under the ik retargeter asset browser i can preview the prerecorded ruth takes.

retargeting animation blueprint

3. we also need to create an animation blueprint for each character that uses a retarget pose from mesh node, under the node details setting the ikretargeter asset to the retargeter we previously created.

hide the source mesh

tick pose

4. nearly there! now create a character blueprint. Inside set the mesh to be the stickman. under its details search for visible and untick all the visibility settings to hide it in the level. now search for tick and set visibility based anim tick option to always tick pose and refresh bones. this will ensure the stickman mesh always updates with its animation even though it is invisible.

add another skeletal mesh and set the skeletal mesh to be to your new character, set its animation class to be the animaton blueprint we just made in step 3.

4. finally, drop the character blueprint into the level. you can set the animation class of the first mesh (the stickman) to either your prerecorded takes anim bp or your livelink mocap anim bp. when the level plays the stickman animation gets retargeted live to yourcharacter mesh.
(This technique also works for radical mocap - they have a custom character that you tweak and their own character IK rig that you retarget to yours, pretty much the exact same method)

openpose -> osc in unreal

run app-win.exe

open a webbrowser and point it to localhost:8000 this is where you shoud see a feed from your webcam overlaid with a dots and stick skeleton.

osc data is streaming from the 17 skeleton points on port 9129
as an osc message looking like this: RECEIVE | ENDPOINT([::ffff:127.0.0.1]:41234) ADDRESS(/pose0) FLOAT(144.01819) FLOAT(72.29978) FLOAT(177.62071) FLOAT(21.599257) FLOAT(148.32224) FLOAT(26.75776) FLOAT(305.67664) FLOAT(32.62451) FLOAT(161.46097) FLOAT(61.923046) FLOAT(443.73434) FLOAT(259.66837) FLOAT(352.28235) FLOAT(269.90558) FLOAT(381.73785) FLOAT(583.20636) FLOAT(266.5818) FLOAT(500.99966) FLOAT(238.74225) FLOAT(514.54865) FLOAT(42.904594) FLOAT(439.00473) FLOAT(434.8785) FLOAT(551.3904) FLOAT(347.69272) FLOAT(510.22787) FLOAT(331.41455) FLOAT(560.9484) FLOAT(260.55704) FLOAT(548.9178) FLOAT(305.8865) FLOAT(551.969) FLOAT(238.51305) FLOAT(502.90475)

below is a table of each skeletal joint against its horizontal, vertical position index:

index horizontal

0
2
4
6
8
10
12
14
16
18
20
22
24
26
28
30
32

index vertical

1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
33

joint name

left ankle
left ear
left elbow
left eye
left hip
left knee
left shoulder
left wrist
nose
right ankle
right ear
right elbow
right eye
right hip
right knee
right shoulder
right wrist

one thing to note: the webbrowser window with the video feed must be the front tab and the browser must not be minimised, else it stops streaming.

second thing to note, in unreal 5.5, when setting it up to receive osc like here the clientipaddress in the createoscserver node must be set to 0.0.0.0. (In the ureal docs it states that for localhost you can use 0 or leave it blank but this wont work in 5.5)

last thing to note, incoming osc horizontal and vertical values seem to be in the ranges 0-640 and 0-480 -- I assume because of the resolution of my web camera feed. In unreal x = horizontal, x = vertical.

set dmx light colour automatically

we want to try setting the colours of dmx controllable theatre lighting automatically. mark coniglio from troikatronix creates an isadora patch for us to do just that. here incoming video is sampled in a small box and then the rgb values are sent over artnet on the local network to a dmx light thats plugged in via its ethernet port. In the image you can see im holding a roll of green tape in the centre of the screen, and this colour is sampled from the video stream and sent both to colour a shape box on top of the video stream, and out as dmx values.

eventurally for the performance we create a multiscene isadora patch that takes live video in from the ai camera and adds some show controls - a fade to black and another fader that blends in the colour square, which is been scaled to tint the entire image. another fader controls a blur effect.

tech setup at trafó

1.
pc 1 runs unreal, showing different scenes audio into the mixer.
the unreal window is captured and runs thru’ isadora which i’m using for show control.

2.
a second pc (pc2) is connected to our ai camera. this video feed also runs thru’ isadora which is also doing show control for the second projector.

3.
after the end of the show, we alt tab between applications to unreal and use radical to get mocap from the video feed to animate avatars in realtime for the audience.

4.
a laptop is used to play music for the show, connected via the mixer into the venue pa system. a second channel mixes in Unreal audio for one of the scenes. a third channel in the mixer is connected to an ipad for the musician to play live after the performance.

tech setup optimised

performative installation version

performative installation version with captury

player, performer installation version with third person viewpoint

a 2-person installation version. ‘expanding the creative possibilites for contemporary dance performances by transposing to a different domain -- into a game world.’

a 'player' uses a game controller to navigate through a series of environments from the nino project featuring motion captured from our trafo performers. the environments are displayed on a large monitor, e.g. 55" 4k or above.

a second audience member acts as a 'performer' and is motion captured by a small camera connected via pc to the internet; this converts their movement into motion capture, which animates an avatar seen onscreen on a large portrait monitor. the two pc's are linked so that the player sees the 'performers' movement.

nino project modina residency