Brainish: Formalizing A Multimodal Language for Intelligence and Consciousness
arXiv Preprint Archive April 14, 2022 Paul Pu Liang
A multimodal inner language called Brainish, combining words, images, audio, and sensations, is proposed as essential for machine consciousness. Building on the Conscious Turing Machine (CTM) model, the authors define Brainish's syntax and semantics and operationalize it through multimodal AI: unimodal encoders segment data, a coordinated representation space composes features for holistic meaning, and decoders produce predictions or raw data. A simple implementation of Brainish is evaluated on multimodal prediction and retrieval tasks using real-world image, text, and audio datasets. The authors argue that such an inner language is crucial for communication and coordination in achieving machine consciousness and intelligence.