OpenAI first unveiled its macOS ChatGPT app in May and apart from the addition of Advanced Voice a few weeks ago, there wasn’t much it could do that you couldn’t do on the web. But the addition of a new ‘app view’ feature, we’re seeing the true value of having a desktop app.
Revealed on Thursday, the updated ChatGPT app for macOS is now able to see the contents of certain apps as you interact with them. Currently limited to developer tools like Xcode and Visual Studio Code, company insiders have told me that it will be able to interact with those applications in the future, essentially allowing you to make changes by talking to ChatGPT and letting it handle the rest.
This is one of the most important upgrades to any AI tool this year and it had stiff competition. So far this year we’ve had powerful new reasoning models, an early computer use concept from Anthropic, and the emergence of natural human-like voice assistants such as Hume AI, GPT-4o dvanced Voice and Gemini Live. That’s before we get into the updates for the best AI image generators and best AI video generators.
But I stand by my assertion. Being able to interact with apps on your computer using nothing but natural language prompts is a game-changing moment for human-computer interaction.
What can you do with ChatGPT + apps?
The new updated app for macOS also comes with a new user interface that puts the ‘app connection’ and ‘model selection’ in a new menu tab above the text entry box. You can use this new app feature to have it review code you’ve written, or even identify issues in the terminal.
Being able to interact with apps on your computer using nothing but natural language prompts is a game-changing moment for human-computer interaction.
If coding isn’t your thing ChatGPT also has access to TextEdit, Apple’s simple but powerful text editor, and you could use this to write very long text-based content and have ChatGPT review it.
The true value isn’t in what it can do now (although it is useful if you write code), but in what it means for the future. Imagine having a Zoom meeting and ChatGPT being able to see the contents of the call and offer live tips on how to respond or interact meaningfully.
Another potentially useful area is in gaming. Picture the scenario: You are in a game of Fortnite and ChatGPT is running in the background. It watches your gameplay and can alert you, in real-time, to any potential threats or offer tips on what to build.
I was able to use ChatGPT’s ability to see Xcode to create a fully functional iPhone ‘endless story’ app where the user enters an idea and an AI creates a story for them that they steer themselves by picking options. It took 25 minutes including troubleshooting.
What comes next?
Working with Code pic.twitter.com/Vt0jaugdTnNovember 14, 2024
The next evolution of the feature will be to have ChatGPT directly interact with apps. This could be a turning point where everyone becomes a developer. It will also make playing games much more fun as the AI will be able to potentially interact with you in-game.
You’d be able to ask ChatGPT to come up with a fun game for iPhone — for argument let's say Flappy Penguin — and the AI will be able to write the code, open Xcode, implement its code suggestions and even handle sending it to Apple for inclusion in the app store. Your involvement is the initial idea and any refinements — all done using natural language.
That is some way off and if we reach that point it is more likely you’d talk to Advanced Voice and have it handle the code and functionality in the background, presenting you with a complete app and report on the screen. Voice is the future of how we interact with machines.
