Participants description

Two groups of subjects participated and followed the protocol entirely: 15 patients with motor deficit (hemiparesia) resulting from acquired brain lesions, and 15 healthy participants were included.

Descriptive information about gender, age, musical background, MBEA scores, and side of hemiparesia are reported in Table 1. More details about each participant are reported on Additional file 1: S3 (Descriptive data of participant’s profiles).

All participants were right-handed, and gender repartition, age and musical background are similar between groups. Motor impairments were evaluated when patients were admitted to the hospital, and were not reliable at the time of the inclusion in the experiment. For time constraints, it was not possible to perform other motor assessments before the experiment. The design of the normalization procedure was aimed at reducing the influence of the different levels of motor impairments. This point will be again specified in the results, discussion and conclusion.

Comparative analysis of MBEA scores revealed lower scores in patient groups than in healthy ones (Additional file 1: S4). This difference is at the limit of significance (test Chi2 p = 0.05).

Movement data resultsIndividual data

Data without normalization of the averages of the total durations performed by each individual with no sound (N1, N2, N3 and N4) and with sonification (Pi, Dr, Md, Mc, Ss), are presented in Fig. 4, considering the subject group (patient and healthy participants) and the arm (paretic side vs. less affected, and dominant vs. non-dominant).

Fig. 4

No sound vs sound condition, for patients and healthy participants. The error bars correspond to the 95% confidence intervals

In Fig. 4, it appears that the average duration of a complete cycle varies across participants. This result must be considered in regards to the fact that we did not give any timing constraint on the movement performance. This is also probably linked to different motor ability, especially in the patients with different levels of motor impairment.

We also observe that there are more variations in the sound conditions compared to the no sound conditions. More precisely, as shown in Additional file 1: S5, the four “no sound” conditions were compared for all participants, and no significant difference was found. Therefore, this stability confirmed that the “no sound” condition can be used for normalizing each participant's sound conditions measurement.

Comparison between the sound and no sound conditions

The comparative analysis of the average of total cycle durations with no sound compared to sonification shows a significant difference for the normalized total time, for each situation considered (p < 0.001 for both arms in patients and healthy participants) (Fig. 5). Specifically, the total average duration increased with sonification compared to cases without sound feedback.

Fig. 5

Mean for the patients (left) and healthy participants (right), for all the sound and no sound conditions, considering the different arms (affected / less affected for the patients, and non-dominant/dominant for the healthy participants). The error bars correspond to the 95% confidence intervals

If we consider in more details the duration of the different phases constituting the extension-flexion cycles of the elbow, we observe significant differences in the average duration of plateau-1 (phase of maximum elongation), retraction phase, and plateau-2 (phase of minimum elongation between two extension phases), when comparing sonification to no sound conditions for both groups. Plateau-1 and the return phase were longer whatever the arm considered both in patients and healthy participants (p < 0.001). Thus, participants remained in maximal extension for a longer time with sonification compared to no sound condition and returned to the starting point slower in the presence of sonification compared to the no sound condition. The average duration of plateau-2 were also longer both in patients for paretic and less-affected arm (p < 0.05 and p < 0.001, respectively) and in healthy participants for non-dominant and dominant arm (p < 0.005 and p < 0.05, respectively).

Concerning the extension phase, significant differences between the sonification and no sound conditions were observed in the healthy participants for both arms and in patients for the less affected arm (p < 0.001).

Comparison between each sound condition

The comparative analysis of the average of total cycle durations according to the sound feedback categories (Anova on repeated measures) shows significant differences between sonifications categories, in the case of the paretic arms of patients and the dominant arms of healthy participants (p < 0.05) (Additional file 1: S6). These significant differences are not found in the other situations (less affected arms of patients and non-dominant arms of controls).

Results concerning the participant’s experienceSound preferences and Experience qualifiers

We report the individual rankings, rated from 1 to 5 points, according to the hierarchical preference of the sound feedback (Table 2) and the qualifying terms respectively (Table 3). The most appreciated sound feedback by both groups of participants is the soundscape, followed by the continuous music. Among the 18 qualifiers list, playful is the first shared term for both groups of participants to qualify their experience.

Table 2 Sound feedback rankingsTable 3 Experience’s qualifiers rankingsThematic analysis based on semi-directive interviews

The thematic analysis revealed six common themes across subject groups.

Reported feelings using the system:

First, the majority of participants reported feeling a difference in the sound context when performing the task (24/30; 14 Patients—10 Healthy), and they preferred performing the gestures in the presence of sonification (29/30; 14P-15H).

Second, the task performance was not considered to be more difficult with sonification than without in the majority of cases (22/30; 11P-11H). Thirteen participants (7P-6H) even reported that it was easier to perform the forearm extension task with sonification. Nevertheless, six participants (2P-4H) reported the experience being occasionally more difficult with some specific couplings, which they justify with two different reasons: a mismatch between the sound type and the gesture to be performed (Md), and when a specific movement quality was required for the sound production (Mc). Concerning the first aspect, several participants underlined that the jerky sound of the discontinuous melody was not matching with the representation of a regular gesture to be carried out. That induced a desire to adapt the gesture in relation to the produced sound, which implied then to perform it in a jerky way. Concerning the second aspect, the participants specified that Mc generates an expectation for the quality of the music produced. In turn, this would require a finer motor control (H01): "Since it’s music, I want it to sound like something fluid that one could listen to".

Finally, one participant mentioned a notable distinction between perception and volition (P04): "The impressions were not different but the intentions could be". This participant reported a deeper involvement in the task performance in the presence of the sonification: "[…] The movement is more voluntary when it produces a sound".

Sound as cues:

Looking more specifically at the interaction between movement parameters and sound coupling, the participants spontaneously mentioned a notion of “cues”, as guides for the movement. This appeared recurrently while mentioning various movement characteristics: amplitude, fluidity, regularity and reproducibility.

Among the characteristics mentioned, temporal aspects were very predominant. Several types of the sonifications effects were described with respect to the temporal characteristics of movement and sound, such as a modulation of the feeling of time (P07): "I had the impression that when I perform a movement with the sound I took more time, I went less quickly to do it", the temporal reference mark (P09): "With music we have a reference point, we keep the same cruising pace", or the more conscious search for an adaptation to the representation conveyed by the sound, in order to obtain a certain sound quality (H08): "When there was no sound I always performed at the same speed, when there was sound I varied the speeds a little because I wanted it to fit with the sound".

Perceived interaction modalities:

For a minority of participants, the interaction modality was unidirectional: four of them felt that the sounds led the movement (4 participants, 3P-1H), and four other participants felt that the gesture controlled the sounds (or vice versa that the sounds followed the gestures, 3P-1H). Other participants (4 patients) expressed having experienced a feedback loop. According to them, the gestures triggered the sounds which in turn provided them with feedback on the gestures, allowing them to adapt to the perceived sound/music.

Finally, for the majority of the participants, the experience of the interaction varied and evolved during the experiment (18 participants; 5P-13H) according to 3 main parameters: the type of sound feedback (1P-6H), the arm performing the task (3P), and the evolution of their understanding of the functioning of the system during the experiment (3H). Regarding the categories, types of sound feedback, and the proposed couplings, participants unanimously expressed that the gestures controlled the sounds for the simple couplings (Pitch and Drum) while the gestures adapted to the sounds for the musical couplings, especially the continuous melody (Mc). Regarding the way the arm performing the task affects the experience of the interaction, patients specified that, for the paretic arm, the gesture controlled the sound, whereas, with the less affected arm, the sound controlled the gesture, or that the gesture adapted to the sounds. Finally, regarding the evolution during the experiment of the interaction understanding, participants expressed that they followed the sound at first, and that later they voluntarily controlled their gesture in order to modulate the sound. H13: "At the beginning I had the impression that I was trying to follow the sound…well to make a gesture following the rhythm, and then I understood that I could control the sound myself with the gesture".

All of these findings suggest that the nature of the sound feedback and the coupling modalities had an influence on the perception of the interaction and on the participants’ experience. H06: “The coupling between the sound and the movement changes the experience of the movement, and so even if you’re trying to do the same movement, even if it’s exactly the same movement, the way you experience it is different, the involvement of the person in the task is really changed”.

Reported emotions

Some participants spontaneously stated that the task was more enjoyable, funnier, more engaging, and more interesting with any type of sonification. Four main affective states were expressed by the participants: playfulness, curiosity, frustration and relaxation. The notion of playfulness is predominant in the spontaneous comments of the participants (8P-9H). Many participants also mentioned their curiosity and surprise at discovering the device. This surprise was often at the origin of the playfulness mentioned above. P07: "I was surprised by the sounds I was making when I was doing the acceleration and deceleration movements. It surprised me, and I liked it, I found it very playful". In other cases, the curiosity was formalized by expecting something from the device. Frustration could also emerge in reaction to the restrictive framework of the instruction: H08: “The fact that I could only do one movement of extension of the arm is a little frustrating because I would have done other movements […] me in any case I wanted to adapt my movements to the sounds”. Finally, the notion of relaxation was expressed many times by the participants, more particularly regarding two couplings: the 'continuous melody' and the ‘soundscape’, implying in some cases body feeling and the task performance. P09: “With the music it softens, it soothes, it’s like we were being massaged, as if we were being put in a second state to be willing. At one point there was music with the sea, the wind, it relaxes you, when you are obliged to make a movement and you can't do it, it relaxes you”.

Mental imagery

Many participants associated the gesture-sound couplings with different mental imagery. The ‘pitch’ was associated with images of a vinyl record, a soft car engine, an ocean or even described as celestial. The 'drum' has been associated with muffled hammering or African drums. The 'discontinuous melody' has been the object of less and contrasted associations (mandolin, stalactic in a cave), although images of bouncing movements have been widely mentioned. The 'continuous melody', which original musical piece was sometimes recognized and named, was associated with the idea of spring, and 'dream-space'. This sonic coupling, in some cases, created the illusion of being a musician (H13): 'I caught myself for thirty seconds as if I were Mozart, so I was very pleased with myself'. Finally, the 'soundscape', a metaphorical space by design, was the most prolific in terms of images, very often associated with the idea of escaping. H09: "There were images that appeared, […] I imagined a kind of walk in a forest, we walk next to the river, then we arrive in a meadow, where there are birds… we imagine the scenery that goes with it".

The stimulation of mental imagery is linked to the participants' preferences: the more the person appreciates the coupling, the more his or her mental imagery is triggered and stimulated. P08: "Every time there was music, I imagined a scene or a moment that I experienced. Especially on the music that I liked in fact ".

Attentional modulation

Evocating the feeling of escape, as well as various emotions, led several participants to report having felt a modulation of their attention during the task, and this depending on the sound context. However, differences across groups should be highlighted.

In the control group, the majority of participants mentioned that their attention was mainly focused on the sounds (10H) H01: "When there was a sound I was thinking less about the movement, I was thinking less about reaching out, I was focusing on the sound". For the other five participants in this group, they could either focus their attention simultaneously or alternately on the sound and the gestures. One participant specified the effects of the feedback loop on their attention and evoked the notion of embodiment: "The attention is not on the movement itself, but on the movement in the context of the effects it has on the music, so I think it changes a lot our way of thinking about the body during the movement".

Within the group of patients, the comments were more contrasted: 4 expressed that their attention was rather focused on the sounds, 4 rather on the gestures, while the others mentioned that the focus of the attention varied, either according to their appreciation of the coupling, or according to the arm performing the task. Indeed, as the gesture could be difficult to perform with their paretic arm, the attention could then shift to the gesture, while being supported by the sound: P06: "[affected side] we are very preoccupied by the very basic movement we have to do. The extension is difficult so we focus on the movement. When I liked the sounds, the attention was directed to the sounds".