ChatGPT’s New Capabilities: Hearing, Seeing, and Speaking

27 Sep 2023 ChatGPT’s New Capabilities: Hearing, Seeing, and Speaking

OpenAI has announced the introduction of new voice and image capabilities in ChatGPT. This enhancement provides users with a more intuitive interface, allowing them to engage in voice conversations with ChatGPT or visually show the AI what they’re discussing. These features are set to revolutionize how users interact with ChatGPT in their daily lives.

For instance, while traveling, users can snap a photo of a landmark and converse with ChatGPT about its significance. At home, they can take pictures of their fridge and pantry, seeking suggestions for dinner and even step-by-step recipes. Moreover, helping a child with homework becomes easier; by taking a photo of a math problem, ChatGPT can provide hints to solve it.

The voice feature will be available on iOS and Android platforms, while the image capability will be accessible across all platforms. The voice functionality is powered by a new text-to-speech model, which can generate human-like audio from text and a brief sample of speech. This advancement was achieved in collaboration with professional voice actors. Additionally, OpenAI’s open-source speech recognition system, Whisper, is used to transcribe spoken words into text.

The image understanding feature is backed by multimodal GPT-3.5 and GPT-4 models. These models apply their language reasoning skills to various images, including photographs, screenshots, and documents containing both text and images.

OpenAI emphasizes a gradual rollout of these features, prioritizing safety and benefits. The organization is aware of the potential risks associated with advanced voice and vision models and is taking measures to ensure responsible usage. For instance, while the new voice technology offers numerous creative and accessibility-focused applications, it also poses risks, such as impersonation or fraud. Therefore, OpenAI is focusing on specific use cases like voice chat.

Vision-based models also come with challenges, from potential hallucinations to high-stakes interpretations. OpenAI has tested these models extensively for risks and has implemented technical measures to limit ChatGPT’s ability to analyze and make direct statements about individuals, respecting privacy.

OpenAI advises users to be aware of the model’s limitations, especially in specialized topics and non-English transcriptions. The organization remains transparent about these limitations and encourages users to verify information from the model.

In the coming weeks, Plus and Enterprise users will have access to these new voice and image features, with other user groups, including developers, gaining access shortly after.

Thought-Provoking Insights:

• The Future of Interaction: With voice and image capabilities, how will the dynamics of human-AI interaction change in the coming years?
• Safety and Ethics: As AI models become more advanced, what measures should be in place to ensure they are used responsibly and ethically?
• Accessibility and Inclusion: How can these new features be harnessed to create more inclusive digital experiences for all users, including those with disabilities?