Run Language Models On Your Smartphone With PocketPal AI

PocketPal AI is an Open source AI Assistant that lets you interact with various small language models (SLMs) directly on your smartphone, even without an internet connection. It offers a number of features such as the ability to download and switch between different SLMs, adjust inference settings for tailored performance, and view real-time performance metrics. PocketPal AI currently supports Android and iOS platforms.

In this blog post, we will discuss the features, benefits, and how to get started with PocketPal AI.

PocketPal AI Features

PocketPal AI comes with a range of features, including:

• Offline Functionality: PocketPal AI's main advantage is its ability to run language models directly on your device without needing internet connectivity. This makes it ideal for use in areas with limited or no internet access.
• Model Flexibility: Users can download and switch between different SLMs, including popular ones like Danube 2 and 3, Phi, Gemma 2, and Qwen. This flexibility allows users to choose the best model for their needs.
• Memory Management: PocketPal AI automatically manages memory by offloading models when the app is in the background and reloading them when you return to the app.
• Customisable Settings: The app allows users to tweak model parameters such as the system prompt, temperature, BOS token, and chat templates.
• Performance Metrics: PocketPal AI displays real-time performance metrics, showing tokens per second and milliseconds per token during AI response generation.

How to Use PocketPal AI on Android and iOS

First, download and install PocketPal AI on your smart phone.

• iOS: Download PocketPal AI from the App Store.
• Android: Get PocketPal AI on Google Play.

Downloading a Model

Once installed, you can download your preferred model. As of writing this, PocketPal AI is pre-configured with the following SLMs:

• Danube 2 and 3
• Phi
• Gemma 2
• Qwen

Please note that you must download your preferred SLM on your mobile to use it. To do so, Open the PocketPal AI app and tap the Menu icon (☰). Go to the Models page. and then select a model from the list and tap Download.

Loading a Model

After downloading, simply tap Load next to the model name to load it into memory.

Chatting with the Model

1. Make sure a model is loaded.
2. Go to the Chat page from the menu.
3. Start typing your questions or prompts!.

Users can access advanced settings by tapping the chevron icon (v) next to a model.

User Experience and Functionality

I don't know how long this app is under development, but the developer of PocketPal AI has open sourced it only on Yesterday.

Users have reported positive experiences with PocketPal AI, praising its speed and user-friendliness. Some users noted faster speeds on iPhones compared to Android devices, possibly due to the use of Apple's Metal GPU API. However, there have been reports of app crashes when loading certain models, particularly on Pixel devices.

One limitation highlighted by users is the difficulty in copying text from the generated responses while preserving formatting, especially Markdown. The developer acknowledged this issue and is looking for solutions.

Conclusion

PocketPal AI seems a powerful and convenient tool that brings the capabilities of language models to your fingertips. Its offline functionality, model flexibility, and user-friendly design make it an excellent choice for anyone who wants to experience AI on their mobile device.

If you have a decent smartphone, install PocketPal AI, do some testing and share your experience with us via the comment section below.

Resource:

• PocketPal AI GitHub Repository