ICMPC14: Dyadic improvisation and mirroring of finger movements

As I unfortunately (or, as the reason is a newborn baby in the family, fortunately) were not able to attend the ICMPC14, I decided to summarise our study (co-authored with Maija Niinisalo and Riitta Hari, a poster was supposed to appear at the Music & Movement 3 session on Thursday, 4:00 pm, Seacliff A-C, Th23) on dyadic improvisation in this blog post. The blog post format allows me to use videos etc. to illustrate the tasks and the data better than a static poster would, and also lets me write a bit more about it than would fit in a poster. If you have any questions, drop me an email. We are currently writing this up as a journal article, so you’ll get the whole story soon, I hope.

Dyadic improvisation and mirroring of finger movements

Summary

We studied kinematics and coordination in a mirror game using fluent, improvised movements (circle drawing & free movement). 32 participants took part in dyads. In turn, one of the participants was appointed the leader, or the dyad was instructed to share leadership. Hand movements were recorded with optical motion capture.

Compared to the leader–follower condition, sharing leadership resulted in more synchronous circles, smoother free movements, and stronger mutual adaptation.

Introduction

Most synchronisation studies investigate rhythmic movements, such as finger tapping, or in non-musical contexts, gait entrainment or synchronisation of heart rates. In these studies, the synchronicity is usually defined as note onset synchrony, where only the simultaneity of the musical events matters, and whatever takes place in between these note onsets is ignored (often not even recorded). Note onset synchrony is of course what matters in musical performance, as different players in the orchestra must match their not onsets at a millisecond accuracy to sound “together”.

Through these kinds of studies, we have learned how the metrical structure of the music generates strong expectations of timing of future events, and how different metrical contexts influence what is considered correct timing. Looking at these behaviours in detail has revealed a very rich set of mechanisms that allow not only the accuracy required for musical performance, but also the combination of  predictability and flexibility that serve improvisation, and the automaticity that make music a universal human behaviour.

We were interested in non-rhythmic, smooth movements,  such as those used in dance, or the kinds of movements musicians make in between, or in addition to, the note onset generating movements. We wanted to see how accurate, predictable, and flexible people would be in coordinating their movements in the (relative) absence of metric structures.

We were inspired by the recent study on mirror game by Noy, Dekel & Alon (2011). Mirror game is a staple in improv theatre. In the game, two people stand facing each other, one starts to move and the other plays their mirror – mirroring all their movements, posture, facial expressions etc. Often the game starts from one person being the mover, and the other their mirror, then switching roles, and finally progressing to a version where there no longer is a leader, but both participants move and mirror each other. To study this fascinating practice scientifically, Noy et al. built a box with two slider buttons, to reduce the mirror game into just one dimension–sliding the button left or right. This contraption  allowed them to measure the movement of the buttons accurately, so that they could measure the behaviours that emerged, as well as eventually make a model of them.

We expanded this one-dimensional mirror game into three, or if you take a more strict view, two dimensions. Our participants were standing up, facing each other as in the “real” mirror game, but instead of having full freedom to do whatever they liked, we asked them to make hand movements using extended arms, and to follow each other’s finger tips. This was a task that was used in a recent hyperscanning paper by Yun, Watanabe & Shimojo (2012).  The short video below shows a snapshot of the task.

Hand movement mirroring from Tommi Himberg on Vimeo.

Methods

We had 32 participants, they performed two kinds of movements: drawing circles and drawing free shapes. Either one of the participants was named the leader, and the other the follower, or they were instructed to share the leadership. Our participants were non-musicians and non-dancers, all were right-handed, were of 19–37 years of age, and 9 of the dyads were female.

As the video above shows, they were standing facing each other, with one arm extended, and they were pointing to each other’s index finger tips. They were positioned so that their finger tips were 5-15 cm apart. They were asked to stand relatively still and perform the movements with extended arms (although remaining relaxed!), thus basically reducing the movement into two dimensions. The depth dimension (anterior-posterior movement) was not actively utilised in the movements, but of course all movements were measured in 3D.

The actual trials were 1 minute long. The video below shows data from the different tasks. I think especially in the free movement, you could relatively easily see who is leading, even without the text spoiling it.

Finger movement examples from Tommi Himberg on Vimeo.

From the 3D position data we calculated the acceleration of the finger tips along their movement trajectory, using the wonderful Motion Capture Toolbox by Petri Toiviainen and Birgitta Burger. We then cross-correlated these acceleration data within dyads to obtain a measure of synchronisation, and the lag at which the maximum sync occurs.

One of the main findings of Noy et al. was that their participants, when sharing leadership, would reach very high synchrony with smooth movements. They called this the “co-confident movement”. These sweet, smooth moves did not have the kinks and stops that would be commonplace in especially in the follower’s movements. The follower would often overshoot, then correct, hesitate a bit etc. Noy et al. analysed this and noticed that this jitter could be found in the 2–3 Hz range, allowing them to measure how much of it there was in the movement, and compare across conditions. We conducted a similar analysis.

Finally, the circle drawing task was chosen for the experiment for the reason that it lies half-way between smooth and rhythmic movement types. The task was to trace a circle with a regular circumference and shape, but the velocity of movement could be varied by the participants. Circle-drawing could be seen as a periodic movement, and it was analysed as such, although the sync measure between players was based on continuous phase difference rather than onsets.

Results

Most of our analyses deal with differences between conditions where one person is the leader and the other one follows, and the shared or joint leadership condition. So first thing for us to check is whether people are moving more or less under these different conditions. Participants moved more (their fingertips travelled a longer distance) in the circles condition than when drawing free shapes, which was expected. Crucially, however, there were no differences in the distance travelled in the different leadership conditions.

The cross-correlations for all free movement trials per condition are in Figure 1. As you can see, whenever one of the participants leads, the other follows with a small lag. The interesting case is the third condition, where the leadership is supposed to be shared. It seems that both follow each other at similar lags as in the other two conditions. These “twin peaks” look very similar to those observed by Ivana Konvalinka and her colleagues in a dyadic finger-tapping study. (The figure for circle drawing is similar, but the correlations are much higher, and the lags smaller, and so all the peaks are much closer together, and the joint leadership is also not two-peaked but a single peak at zero.)

Cross-correlation - free shapes

Cross-correlation of free shape drawing. Blue line: A leads, red line: B leads, black line: joint leadership.

This correlation is for the whole figure. The question is, then, how does this two-peaked correlation form over time? Does one participant take the leader role for the first half of the trial, then they switch roles and the other participant leads until the end? Or do they take shorter shifts? To answer this question, we conducted the cross-correlation in a moving window. Below in figure 2, the cross-correlogram for our pair 3 is presented.

Cross-correlation over time for an individual pair, shared leadership. Bright colours indicate high correlation coefficients. A leads when bright colours appear on top half of the figure.

Cross-correlation over time for an individual pair, shared leadership. Bright colours indicate high correlation coefficients. Participant A leads when bright colours appear on top half of the figure.

 

According to the cross-correlogram, the answer seems to be that both are following each other, and both are leading at the same time. They are mutually adapting to each other, continuously. This is the mechanism through which synchrony and coordination is achieved, and judging by the similarity between our results and those of Konvalinka et al., this seems to hold in both rhythmic and non-rhythmic contexts.

So, how about that co-confidence? We calculated the amount of energy at different frequencies of movement for the free movement task, and ran an ANOVA to check whether the different leadership conditions have an effect. Indeed they did.

Jitter in different conditions. The difference between Following and Joint Leadership is significant at p<0.01 level.

Jitter in different conditions. The difference between Following and Joint Leadership is significant at p

Figure 3 shows the amount of jitter (movement at 2–3 Hz) in different conditions, for the free movement task. Overall, the joint leadership task is the one with lowest amount of jitter, although the difference between sharing the leadership and being leader alone is not statistically significant. Participants were most jittery when following the other, as expected, and the difference between following and sharing leadership was highly significant.

For the circle drawing task, we also conducted a phase synchronisation analysis. We obtained a time-series of the instantaneous phase of the movement using Hilbert transform, and then looked at the phase difference between participants in the dyads. As evident already in the video above, participants were able to do this very well, regardless of the condition. Figure 4 below shows the average phase differences for each trial as a polar plot.

Polar graph of the phase differences for circle-drawing trials. Angle represents lag, the distance from the origin represents synchrony, ranging from 0 (no sync) to 1 (perfect unison).

Polar graph of the phase differences for circle-drawing trials. Angle represents lag, the distance from the origin represents synchrony, ranging from 0 (no sync) to 1 (perfect unison).

The dots represent the concentration measure R for the distribution of phase errors for each trial. R is a measure ranging from 0 to 1. The polar plot in figure 4 also shows the average phase difference.  So, a perfectly synchronous, zero-lag performance would be a dot at the zero-angle spot on the circumference of the circle. All trials were very close to this ideal case. We ran a t-test to see if the leadership conditions had an effect, and found a very small but significant difference in favour of the joint leadership task (figure 5 below).

Synchronisation (concentration measure R) in circle-drawing task.

Synchronisation (concentration measure R) in circle-drawing task.

 

 

 

 

 

 

 

 

 

 

 

 

Conclusions

In our study we extended the mirror game paradigm from one dimension (Noy et al., 2011) to 3D, and used motion capture to analyse synchrony in a free and circle drawing tasks. Our participants were non-specialists, but had no trouble in performing these tasks with high accuracy and synchronicity, and even in a co-confident way. Participants in Noy’s experiment were experienced improvisers, and the authors have told me, people with no experience in improv had trouble reaching the co-confident state. Perhaps it is because in their task, the interaction between participants is mediated by the box and the sliding buttons. In our task, participants are interacting directly, although not touching each other. Some participants even said in debriefing that they felt they were in contact with their partner, either so that they were both holding an object in between their fingers (which they then tried not to drop), or their fingers were actually elongated and touching each other.

In our study, participants in the joint leadership task reached a higher level of synchrony, smoother movements and stronger mutual adaptation than when they had leader-follower roles assigned.

References

Konvalinka, I., Vuust, P., Roepstorff, A., & Frith, C. D. (2010). Follow you, follow me: continuous mutual prediction and adaptation in joint tapping. The Quarterly Journal of Experimental Psychology, 63(11), 2220–2230. http://doi.org/10.1080/17470218.2010.497843

Noy, L., Dekel, E., & Alon, U. (2011). The mirror game as a paradigm for studying the dynamics of two people improvising motion together. Proceedings of the National Academy of Sciences, 108(52), 20947–20952. http://doi.org/10.1073/pnas.1108155108

Yun, K., Watanabe, K., & Shimojo, S. (2012). Interpersonal body and neural synchronization as a marker of implicit social interaction. Scientific Reports, 2. http://doi.org/10.1038/srep00959

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s