| 其他摘要 | Category learning refers to the process by which individuals classify cluttered information and experiences into meaningful categories and concepts that allows individuals to flexibly adapt to their environment and take action (Seger & Miller, 2010). Previous researches on category learning have explored how people extract statistical correlations between unisensory representations to acquire new multisensory representations or amodal representations, but few studies have examined how category learning affects the multisensory representations that have been formed. Multisensory or amodal representations refer.坦abstract knowledge representations generated by individuals when integrating or processing information, which are independent of the surface features of stimuli, and independent of sensory modalities. Currently, a large number of studies have shown crossmodal correspondence across stimuli and stimulus dimensions, but it is not clear whether the crossmodal correspondence is based on multisensory representations, and whether and how category learning affects crossmodal correspondence. Therefore, this study combines a category learning task and a crossmodal matching task to investigate the types of representations underlying crossmodal correspondence and the time course of crossmodal transfer, and to reveal the role of category and semantic labeling in the formation of multisensory representations.
Study 1 combined category learning and crossmodal transfer tasks to investigate the types of multisensory representations on which crossmodal correspondence is based, and to examine how category knowledge affects multisensory representations already present in the brain. The results of Experiment 1 found that the acquired category knowledge about visual position influenced the categorization of pitch when category learning was performed under the visual only condition, indicating that crossmodal transfer occurred between visual position and auditory pitch. Moreover, the correspondence of category labels for position and pitch in multisensory category learning did not affect the crossmodal correspondence effect of pitch-position. The results of Experiment 2 showed that the acquired category knowledge about visual size did not affect the categorization of pitch, but interestingly, the way that combined category labels for size and pitch in multisensory category learning determined the direction of pitch-size crossmodal correspondence, indicating that pitch-size crossmodal correspondence is not stable and can be affected by category and semantic labels. Experiment 3 balanced the difference in task difficulty between position and size category judgments, and further found that no crossmodal transfer occurred between size and pitch, and the way that category labels of size and pitch corresponded in multisensory category learning influenced the direction of pitch-size correspondence. Experiment 4 provided additional evidence that crossmodal transfer between pitch and size did not occur because there was no stable pitch-size correspondence.
Study 2 examined the effects of category learning on multisensory representations by manipulating the amount of training in a multisensory category learning task. The correspondence between pitch and position category labels learned by participants in Experiments 5 and 6 was the opposite of the crossmodal correspondence between pitch and position, with 20 blocks completed at once in Experiment 5, and 12 blocks completed every day for seven days, for a total of 84 blocks in Experiment 6. It was found that in both Experiment 5 and Experiment 6, the pitch-position correspondence between audiovisual stimuli did not affect the acquisition of category labels with opposite pitch and position correspondence across modalities in multisensory category learning, and the accuracy of simultaneous audiovisual presentation during learning was significantly higher than that of separate presentation, indicating the classical multisensory facilitation effect. More importantly, the experimental results showed that prolonged the training of the inverse relationship of category labels could change but could not reverse the direction of pitch-position matching. These results suggest that the pitch-position correspondence tends to be based on structural associations and statistical coupling and semantic mediation effects cannot fully explain this phenomenon.
Study 3 used the event-related potential technique to examine the dynamic time course of crossmodal correspondence affecting the transfer of category knowledge across modalities. Experiment 7 replicated the behavioral results of Experiment 1，confirming that when there is pitch-position crossmodal correspondence between audiovisual stimuli, individuals can spontaneously transfer categorization knowledge about visual position to auditory pitch, indicting crossmodal transfer effects of category knowledge. ERP results showed that in the auditory categorization task, auditory stimuli induced a greater N1 effect, and a larger P2 effect was induced in the post-learning than in the pre-learning test. These results suggest that the effect of crossmodal correspondence on the crossmodal transfer of category knowledge emerges in the early feature integration phase rather than the late category judgment phase.
This study systematically investigated the effects of crossmodal correspondence on the crossmodal transfer of category knowledge and its time course, and found that crossmodal correspondence is based on modality-independent multisensory representations. Simultaneously, this study also examined the effects of category learning on crossmodal correspondence, and found that multisensory category learning can determine the direction of the pitch-size correspondence, and can change but cannot fully invert the already formed pitch-position correspondence. These findings deepen the understanding of the mechanism of crossmodal correspondence generation, enrich the theory related to multisensory representation, and help to solve the problem of crossmodal transfer in category learning. At the same time, the related research results have implications for the setup of sensory substitution devices. |
修改评论