Google’s ‘AppFunctions’ Lets Gemini Actually Do Things Inside Your Android Apps

 

# Google’s ‘AppFunctions’ Lets Gemini Actually Do Things Inside Your Android Apps

**Published: February 26, 2026**

You know that feeling when you're bouncing between five different apps just to get one simple thing done?

Find a recipe in an email, switch to a notes app to write down the ingredients, open a grocery app to add them to your cart, then jump to your calendar to schedule dinner.

It's exhausting. And frankly, it's ridiculous that our phones can't just... connect the dots for us.

Google thinks they've finally solved this. The company just detailed a new Android 16 feature called **AppFunctions** that lets Gemini reach directly into your apps and perform specific tasks on your behalf. Think of it as giving your AI assistant a set of keys to actually *do* things inside your apps, not just open them .

Let me walk you through what AppFunctions actually does, how it compares to the "UI automation" approach also launching soon, and what this means for the future of using your phone.

---

## The Short Version

**What happened:** Google fully detailed AppFunctions, an Android 16 platform feature and Jetpack library that lets apps expose specific functions for AI assistants like Gemini to access and execute on your device .

**How it's different:** Unlike the UI automation approach announced for Pixel 10 and Galaxy S26 (which literally watches and taps your screen), AppFunctions is a structured, developer-built framework where apps declare exactly what Gemini can do—like "create a task" or "search photos."

**Real-world example:** On the Galaxy S26, you can now ask Gemini to "Show me pictures of my cat from Samsung Gallery." Gemini triggers a specific function in Samsung Gallery and displays the results right inside the Gemini app—no manual scrolling required .

**What's coming:** Android 17 will expand these capabilities, and Google is already working with select developers to build more integrations .

**The bigger picture:** Android is evolving from an operating system you navigate to one where you simply tell an AI what you want done, and it handles the rest .

---

## What Is AppFunctions? The Structured Approach

Let's start with the basics, because "AppFunctions" sounds technical but the concept is actually pretty straightforward.

**AppFunctions is an Android 16 platform feature and Jetpack library** that allows app developers to "expose specific functions for callers, such as agent apps, to access and execute on device" .

Think of it like this: instead of Gemini trying to figure out how to use an app by looking at its screen (which is what UI automation does), the app itself says "here are the things I know how to do—create a task, search photos, add a calendar event." Gemini can then call those functions directly, like using a remote control instead of reaching through the screen.

**Google equates AppFunctions to the Model Context Protocol (MCP)** that's popular for agents and server-side tools, but with a key difference: these functions happen **locally on the Android device** .

### Why This Matters

- **Precision:** The app defines exactly what Gemini can do, so there's no guesswork

- **Privacy:** Functions execute on-device, not in the cloud

- **Speed:** Direct function calls are faster than simulating taps and scrolling

- **Reliability:** No worrying about UI changes breaking the automation

---

## Real-World Examples: What AppFunctions Can Actually Do

Google shared several concrete examples of how AppFunctions will work in practice. These aren't hypothetical—they're being built right now.

**Table 1: AppFunctions in Action**

| **Category** | **User Request** | **What AppFunctions Does** |

| :--- | :--- | :--- |

| Task Management | "Remind me to pick up my package at work today at 5 PM" | Identifies the relevant task management app and invokes a function to create a task, automatically populating title, time, and location fields based on the user's prompt . |

| Media & Entertainment | "Create a new playlist with the top jazz albums from this year" | Executes a playlist creation function within a music app, passing context like "top jazz albums for 2026" as the query to generate and launch the content immediately . |

| Cross-App Workflows | "Find the noodle recipe from Lisa's email and add the ingredients to my shopping list" | Uses an email app's search function to retrieve the content, extracts ingredients, and invokes a shopping list app's function to populate the user's list—all without leaving the Gemini interface . |

| Calendar & Scheduling | "Add Mom's birthday party to my calendar for next Monday at 6 PM" | Invokes the calendar app's "create event" function, parsing "next Monday" and "6 PM" to create the entry without manually opening the calendar . |

**The key insight:** In all these examples, the user never leaves Gemini. The AI handles the entire workflow in the background, presenting only the results.

---

## The Samsung Gallery Integration: A Live Example

The most visible implementation of AppFunctions right now is with **Samsung Gallery on the Galaxy S26 series** .

Instead of manually scrolling through photo albums to find that one picture of your cat, you can simply ask Gemini: "Show me pictures of my cat from Samsung Gallery."

Here's what happens behind the scenes:

1. Gemini receives your voice or text query

2. It intelligently identifies that Samsung Gallery has a photo search function

3. It triggers that specific AppFunction, passing "cat" as the search parameter

4. Samsung Gallery executes the search locally and returns the photos

5. Gemini displays the results directly in the Gemini app

The experience is multimodal—you can use voice or text. And once the photos appear, you can even use them in follow-up conversations, like sending them to friends in a text message, all without ever leaving Gemini .

This integration is coming to Samsung devices running OneUI 8.5 and higher .

---

## What Google Apps Already Support

AppFunctions isn't starting from scratch. Google says the Gemini app is **already using AppFunctions** to power its Calendar, Notes, and Tasks integrations in Google apps and OEM defaults .

That means right now, if you're on a supported device, Gemini can:

- Create calendar events

- Add and manage notes

- Handle task creation and updates

These are the foundation, and Google is building from there.

---

## The Second Approach: UI Automation for Everything Else

AppFunctions is elegant, but it has a limitation: it only works when developers explicitly build the integrations.

What about all the apps that don't have AppFunctions yet?

That's where Google's second approach comes in: **UI automation**.

### What UI Automation Does

Google is "developing a UI automation framework for AI agents and assistants to intelligently execute generic tasks on users' installed apps" .

Instead of calling specific functions, this approach lets Gemini actually **see and interact with your screen**. It can scroll, tap, type, and navigate through apps just like you would—but automatically.

**The key difference:** This is "the platform doing the heavy lifting, so developers can get agentic reach with zero code" .

### What UI Automation Can Do

At launch, UI automation will support select apps in three categories:

- Food delivery

- Grocery

- Rideshare

**Example tasks:**

- "Book a ride home" – Gemini enters the location, picks a ride type, sets the pickup time

- "Reorder my last meal" – Gemini navigates through a food delivery app and places your usual order

- "Add items to my grocery cart" – Gemini builds a cart based on your shopping list or previous orders

### How It Works Under the Hood

When you ask Gemini to handle a task, it runs the application in a **"secure, virtual window on your phone"** . Importantly, it cannot access the rest of your device—only that virtual screen .

What's happening in that window is processed in the cloud, but you can view the progress in real-time. A notification lets you see Gemini scrolling, tapping, and typing. You can continue using your phone for other things while Gemini works in the background .

### Safety and Control Features

Google has built several safeguards into UI automation:

**Table 2: UI Automation Safety Features**

| **Feature** | **What It Does** |

| :--- | :--- |

| Real-time monitoring | You can view Gemini's progress through a secure window or notifications . |

| Manual takeover | You can "Take control" at any moment if something looks off . |

| Explicit confirmation | Gemini prompts you to open the app to tap the actual "buy" or "order" button—you're always the one who completes financial transactions . |

| Granular permissions | You must explicitly grant permission before Gemini automation can run . |

| Clear start/stop | Automations begin with your command and stop as soon as the task is finished . |

---

## Which Devices Get What

Both AppFunctions and UI automation are rolling out now, but availability varies.

**Table 3: Feature Availability**

| **Feature** | **Devices** | **Timing** |

| :--- | :--- | :--- |

| Samsung Gallery + Gemini | Galaxy S26 series, OneUI 8.5+ devices | Now  |

| UI Automation (beta) | Galaxy S26 series, Pixel 10 series | March 2026 (US & South Korea only)  |

| AppFunctions for developers | Android 16+ devices | Now (early beta)  |

| Expanded capabilities | Android 17+ devices | Future  |

**Supported app categories at launch:** Food delivery, grocery, and rideshare apps including Uber, Doordash, and Grubhub .

**Regional availability:** The UI automation beta starts in the United States and South Korea .

---

## The Developer Perspective: Building for the Agentic Future

For developers, AppFunctions represents a new way of thinking about apps. Instead of building for human thumbs, you're building for AI agents.

Google describes it as "introducing early stage developer capabilities that bridge the gap between your apps and agentic apps and personalized assistants, such as Google Gemini" .

### What Developers Need to Know

1. **AppFunctions is an Android 16 platform feature** with an accompanying Jetpack library .

2. Developers "detail their app's capabilities as tools that agents and AI assistants can use" .

3. These functions happen **locally on the Android device**, not in the cloud .

4. Google is "designing these features with privacy and security at their core" .

5. The UI automation framework offers a **zero-code option** for developers who don't build direct integrations .

### Looking Ahead to Android 17

Google says Android 17 will "broaden these capabilities to reach even more users, developers, and device manufacturers" .

The company is currently "building experiences with a small set of app developers, focusing on high-quality user experiences as the ecosystem evolves" .

---

## Privacy and Security: Google's Approach

Whenever you give an AI access to your apps, privacy concerns are top of mind. Google seems aware of this and has built multiple layers of protection.

**On-device processing for AppFunctions:** Because AppFunctions run locally on your device, sensitive data doesn't leave your phone .

**Secure virtual window for UI automation:** When Gemini needs cloud processing for UI automation, it runs in a "secure, virtual window" that cannot access the rest of your device .

**User control throughout:** You can monitor progress, take over at any time, and must explicitly confirm sensitive actions like purchases .

**Matthew McCullough**, VP of Product Management for Android Development, framed it as a shift toward a "task-focused model" where agents execute actions across applications, but always with user transparency and control .

---

## What This Means for You

### If You're a Galaxy S26 or Pixel 10 Owner

You're getting these features first. Starting in March, you'll be able to offload tedious tasks to Gemini—booking rides, reordering food, building grocery carts. And if you have a Galaxy S26, you can already ask Gemini to find specific photos in Samsung Gallery.

### If You're on Older Android Hardware

You'll have to wait. AppFunctions is built into Android 16, and the UI automation beta is launching on new devices. But Google's track record suggests these capabilities will eventually trickle down.

### If You're a Developer

This is your moment to start thinking about how your app can work with agents. The early beta is open to a small group of developers now, and broader access is coming later this year .

### If You're Just a Normal Person

Your phone is about to get a lot smarter. The days of manually jumping between apps to complete simple workflows are numbered. Soon, you'll just tell your phone what you want done, and it will handle the rest.

**Sameer Samat**, Google's Android ecosystem president, put it well: Android is evolving from a traditional operating system to one that "truly understands you and serves you" . He calls it "agentic AI" but summarizes it more simply as "getting things done" .

---

## The Bigger Picture: Where Android Is Headed

This isn't just a feature update. It's a fundamental shift in how we interact with our phones.

For the past 15 years, smartphones have been about **apps**—grids of icons you tap to open, then navigate manually. It's been the same paradigm since the iPhone launched in 2007.

AppFunctions and UI automation represent a move toward a **task-based model**. Instead of thinking "I need to open my calendar app," you think "I need to add an event." The OS figures out the rest.

**McCullough** described it as a shift where "agents execute actions across applications" . The app grid doesn't disappear, but it fades into the background. Your primary interface becomes natural language.

This is what tech companies have been promising for years. With AppFunctions and the Gemini automation features landing on actual devices, it's finally starting to feel real.

---

## Frequently Asked Questions

**Q: What's the difference between AppFunctions and UI automation?**

A: AppFunctions is a structured framework where apps explicitly declare functions for Gemini to call—like "create task" or "search photos." UI automation is a more general approach where Gemini can see and interact with your screen, scrolling and tapping like a human would .

**Q: When can I use these features?**

A: The Samsung Gallery integration is available now on Galaxy S26 devices. The UI automation beta for food, grocery, and rideshare apps launches in March on Galaxy S26 and Pixel 10 series, initially in the US and South Korea .

**Q: What apps are supported at launch?**

A: For UI automation: select apps in food delivery, grocery, and rideshare categories, including Uber, Doordash, and Grubhub . For AppFunctions: Samsung Gallery, plus Google's own Calendar, Notes, and Tasks .

**Q: Is my data private?**

A: AppFunctions runs locally on your device. UI automation runs in a "secure virtual window" that can't access the rest of your phone. You can monitor progress in real-time and must confirm sensitive actions like purchases .

**Q: Can I stop Gemini mid-task?**

A: Yes. You can "Take control" at any moment, and notifications let you monitor what Gemini is doing .

**Q: Will this work on my older phone?**

A: AppFunctions requires Android 16. The UI automation beta is launching on new devices (Galaxy S26 and Pixel 10). Older devices may get features eventually, but timing is unclear.

**Q: Do developers have to do anything for this to work?**

A: For AppFunctions, yes—developers need to use the Jetpack library to expose functions. For UI automation, no—it works with existing apps, though performance may vary .

**Q: What about apps I don't want Gemini to access?**

A: You control permissions. Automations can't begin without your command, and you can revoke access at any time .

**Q: Will this replace apps entirely?**

A: No. Apps will still exist, but the way you interact with them changes. Instead of manually navigating, you tell an AI what you want done, and it handles the execution.

**Q: When is Android 17 coming with expanded features?**

A: Google hasn't announced a timeline, but typically major Android versions arrive in the fall. More details are expected later this year .

---

## The Bottom Line

Here's what I keep coming back to.

For years, we've heard about AI assistants that can "do things" for us. But mostly, that meant answering questions or maybe setting a timer. The real promise—having an AI that can actually navigate apps and complete tasks—always felt just out of reach.

AppFunctions and the new Gemini automation features change that.

On the Galaxy S26, you can already ask for photos and have Gemini pull them from Samsung Gallery without you lifting a finger. Soon, on both Galaxy S26 and Pixel 10, you'll be able to book rides, order food, and build grocery carts just by asking.

This is the agentic future tech companies have been promising. And for the first time, it's landing on real devices that real people can buy.

**Samir Samat** put it in perspective: this is about Android evolving into an operating system that "truly understands you and serves you" . He calls it "agentic AI" but prefers a simpler description: "getting things done."

For anyone who's ever wasted 10 minutes jumping between apps to complete a simple task, that's exactly what we've been waiting for.

---

*Got thoughts on Gemini automation? Tried it on your Galaxy S26 or Pixel 10? Drop a comment and let me know.*