- Microsoft is seriously expanding the powers of the Copilot app – for all Windows 11 PCs
- Copilot Voice will provide a new method of firing up the AI with voice input, backed by a global rollout of Copilot Vision
- Broader AI powers are coming including Copilot Actions that allow AI to interact with the files on your PC
Microsoft teased that something big was happening today, and now we’ve heard all about its plans to make voice interaction a bigger part of the Windows 11 experience – and how AI functionality will be supercharged in the OS.
The expectation was that this announcement would be focused on AI, and indeed it is, but the twist is that this isn’t just for Copilot+ PCs. Microsoft is promising a “new wave of updates that make every Windows 11 PC an AI PC – with Copilot at the center of it all”.
As Microsoft’s Yusuf Mehdi, VP and Consumer Chief Marketing Officer, makes clear in a blog post, new voice commands are coming to Windows 11 in the form of Copilot Voice, backed with a wider global rollout of Copilot Vision – and the ability for the Copilot app to take more actions on your behalf.
Let’s start with the voice element first – all you need to do is say “Hey Copilot” (and definitely not “Hey Cortana”) to invoke the AI. The Copilot microphone will appear on your screen, and you can issue voice commands or pose a query to the AI. Saying “Goodbye” or closing the panel will end the conversation automatically (with audible chimes confirming both the start of a session, and the end).
I should note that you need to opt-in for this voice functionality on the Windows 11 desktop (by enabling it in the settings for the Copilot app).
Behind Copilot Voice we have the impending global rollout of Copilot Vision, which has only been available in the US so far.
The Vision feature – again, given your opt-in permission – analyzes the content of your screen and can provide help in terms of instructing you on how to use apps, or achieve certain tasks in Windows 11. It can even help with troubleshooting when things have gone wrong in the OS, or provide tips in a game. All of this can be facilitated with your voice, given Copilot Voice and Vision working in tandem.
All of which is well and good, but what does AI actually do for you in Windows 11? Microsoft further explains that it has some nifty new powers inbound, which will be in testing (preview releases) soon.
Those new AI powers
First up, the Windows 11 taskbar will get an Ask Copilot box, complete with one-click access to Copilot Voice and Vision via their icons (goggles and a mic) – should you enable this. We’re also promised a faster, better Windows search with this new functionality (it’s about time – and notably this doesn’t involve giving Copilot access to your files, nothing is changed in that respect under the hood).
However, Microsoft’s big move with AI (and agents) coming to testing is the introduction of Copilot Actions for the files on your PC.
If Copilot Actions rings a bell, it’s because this is something Microsoft has previously introduced for the web, meaning the AI can go online for you to complete tasks such as making a hotel booking, or buying tickets for a concert.
In a new ‘experimental mode’ in Copilot Labs for testers, Microsoft will allow for AI Actions to be taken with files on the local drive in Windows 11. That means you can get AI to sort through your recent vacation photos, for example, or pull some specific info from a document.
You just need to describe what you want doing (using your voice, if you prefer), and the AI agent will “attempt to complete it by interacting with desktop and web applications”, Microsoft informs us.
You can leave the AI to it, and pop back to check on its progress as and when you wish. The potential for AI making mistakes is always present, of course, and Microsoft underlines that “real-world testing of this experience is critical” because of this – and that it’ll be starting with a “narrow set of use cases” in a slow and cautious manner.
Microsoft underlines that security will clearly be a vital consideration here, and defending against new threats that such a system would introduce, such as cross-prompt injection attacks which could override instructions to AI agents.
Microsoft says it’s establishing a “set of durable security and privacy principles that must be met to make use of new agentic capabilities in Windows”, and that includes limiting agentic privileges and permissions (to files you allow the AI to work with).
As well as Copilot Actions, Microsoft has a new general AI agent for Windows 11 called Manus, which can tackle complex tasks, with the software giant explaining that it can create a website for you. Using the Manus AI action in File Explorer, you can gather a set of documents and images in a folder, and get Manus to build a website based on that material – which sounds seriously impressive.
Microsoft is also bringing in more Copilot connectors, in order to link services like Gmail to the Copilot app, so you can ask the AI about your upcoming appointments logged with Google, for example. The Copilot app will also be able to help fine-tune your PC settings, as Microsoft has already revealed.
Analysis: A big step forward for AI
All of this is in the works, of course, and not even in testing yet – so much of what we’re talking about here could be some way off.
Particularly the idea of letting Copilot loose to work with your files on the desktop, which Microsoft needs to be very careful with the implementation of, as noted on the security front. Don’t underestimate what Microsoft is planning here. The idea of Copilot Actions is that it’s an AI agent which takes on tasks you give it and completes them by “interacting with your apps and files, using vision and advanced reasoning to click, type and scroll like a human would.”
This is partly what Microsoft means by using your mouse and keyboard a lot less. It isn’t just about giving voice commands in Windows 11, or voice queries to the Copilot app, it’s also about the AI actually doing stuff for you in the operating system: Voice, Vision, and Actions.
Microsoft is certainly emphasizing how important voice input is though, noting that the more detail and context you share in an AI query to Copilot, the better the response – but that typing it all out is a pain.
Mehdi observes that because of this, when people use voice, “they engage with Copilot twice as much as when they use text”, and the new wake word (Hey Copilot) is a way of making interacting with Copilot even easier on the desktop.
Mehdi underlines: “We believe this shift to conversational input will be as transformative as the mouse and keyboard in terms of unlocking new capabilities on the PC for the broadest set of people.”
It’s certainly clear that this is a big moment for Windows 11. Thus far, when it comes to AI, much of the criticism is about the features therein being underwhelming – but the powers covered here are more far-reaching and potentially impressive. As is the scope of the abilities coming to the Copilot app broadly for all PCs, not just Copilot+ laptops.
You might also like
https://cdn.mos.cms.futurecdn.net/VpDU3zwnzbGKAaKDHPjVa5-1565-80.jpg
Source link
