ChatGPT Introduces Integration With Third-Party Apps For Mac Users, Here Are The Details

The feature, called "Work with Apps," is still in its early stages and not yet an AI agent. OpenAI describes this integration as a foundational step toward building agentic systems.

OpenAI’s ChatGPT is beginning to integrate with other apps on your computer, TechCrunch reported. On Thursday, the company announced that its ChatGPT desktop app for macOS can now interact with several developer-focused tools, including VS Code, Xcode, TextEdit, Terminal, and iTerm2. This update eliminates the need for developers to copy and paste their code into ChatGPT manually. When enabled, the feature automatically sends the code you’re working on, along with your prompt, directly to the chatbot for contextual understanding.

Unlike popular AI tools like GitHub Copilot or Cursor, ChatGPT currently doesn’t have the capability to write code directly into developer apps. The feature, called "Work with Apps," is still in its early stages and not yet an AI agent. OpenAI describes this integration as a foundational step toward building agentic systems. A significant hurdle for AI agents remains their ability to comprehend the broader context of a user’s computer screen beyond just prompts and responses.

OpenAI is initially focusing this feature on coding apps, as AI-powered coding tools have emerged as one of the most popular applications for large language models (LLMs). The functionality is currently available to Plus and Teams users, with plans to expand it to Enterprise and Edu users in the coming weeks. OpenAI also plans to extend compatibility to other types of apps, particularly text-based tools that support writing tasks.

OpenAI Demonstrates

In a demonstration with TechCrunch, an OpenAI employee showcased the feature using the ChatGPT app alongside an Xcode project simulating the solar system. The project was incomplete, missing Earth. By selecting an Xcode tab within the ChatGPT app, the user instructed the AI to analyze the Xcode environment and prompted it to “add the missing planets.” ChatGPT successfully generated code to represent Earth in a format consistent with the project. However, the generated code still needed to be manually copied and pasted back into the development environment.

To enable interaction with various apps, OpenAI primarily relies on macOS’s accessibility API, which reads text from applications and translates it into input for ChatGPT. This approach leverages the macOS screen reader technology, originally designed for the VoiceOver feature, which has been a reliable tool for many common applications over nearly two decades, said OpenAI desktop product lead Alexander Embiricos. However, its functionality may vary depending on the app.

For certain applications, like Microsoft’s VS Code, the "Work with Apps" feature requires users to install a specific extension to query content. However, Apple’s screen reader, which underpins the feature, has its limitations—it can only process text and cannot interpret visual elements such as images, object layouts, or videos. For some supported apps, the feature sends the last 200 lines of code to ChatGPT along with each prompt, while for others, it uses all the code in the active window as input. Users can highlight specific sections to guide the chatbot’s focus, but the surrounding text will also be included in the context. This process is likely to consume a significant number of input tokens.

How OpenAI plans to expand this capability to apps that aren’t compatible with Apple’s screen reader remains uncertain. Competitor Anthropic has introduced an alternative approach, utilising AI to analyse desktop screenshots for understanding and interacting with other applications. While this method shows potential as a more general-purpose solution not reliant on APIs, it currently suffers from significant drawbacks, including slow performance and frequent errors. Despite its imperfections, Anthropic’s strategy demonstrates a broader vision for AI agents beyond simple text-based interactions.

Ebiricos told TechCrunch, “This isn’t meant to be an agent; it’s a way to collaborate with coding tools to start, and there will be more tools coming soon. On the side of agents, I think this is a really key building block. This idea that ChatGPT understands or can work with all the content that you have so that it can help with it.”