Research

The advent of large language models (LLMs) has opened up new opportunities in the field of mobile task automation. Their superior language understanding and reasoning capabilities allow users to automate complex and repetitive tasks. However, due to the inherent unreliability and high operational cost of LLMs, their practical applicability is quite limited. To address these issues, this paper introduces MobileGPT, an innovative LLM-based mobile task automator enhanced with a unique app memory. MobileGPT emulates the cognitive process of humans interacting with a mobile app—explore, select, derive, and recall. This approach allows for a more precise and efficient learning of a task’s procedure by breaking it down into smaller, modular components that can be re-used, re-arranged, and adapted for various objectives. We implement MobileGPT using online LLMs services (GPT-3.5 and GPT-4) and evaluate its performance on 50 unique mobile tasks across 5 widely used mobile apps. The results indicate that MobileGPT can adapt learned tasks to varying contexts with 100% accuracy and reduces their latency and cost by 69.22% and 77.36% compared to a GPT-4 powered baseline.

Deep Neural Networks (DNNs) are essential in IoT and CPS applications, with hardware accelerators like Google Edge TPU enhancing performance and energy efficiency. However, unpredictable execution times limit real-time deployment due to B1) deadline-oblivious SRAM allocation and B2) non-preemptive scheduling. Experiments show up to 5.2x execution time variation caused by these issues. To address this, we propose SPET, a system for transparent SRAM allocation and model partitioning. SPET offers i) flexible SRAM allocation without TPU compiler modifications, ii) adaptive model partitioning for predictable preemption, and iii) priority-based online scheduling. Evaluation on TensorFlow Lite with Google Coral shows SPET improves schedulability by up to 79%. A case study on emergency braking in a self-driving car highlights its safety benefits through timely DNN inference.

Mobile apps offer diverse features to enhance user experience, yet users often struggle to use multiple apps simultaneously on small smartphone screens. This paper introduces A-Mash, a mobile platform that simplifies concurrent app interactions to the ease of using a single app. A-Mash allows users to mash up UIs of different apps on one screen based on their preferences by 1) extracting UIs from unmodified apps (dynamic UI extraction), 2) embedding these UIs into a single wrapper app (cross-process UI embedding), and 3) ensuring the process remains transparent to users. To our knowledge, A-Mash is the first system to seamlessly integrate UIs of unmodified legacy apps on a single screen, creating the illusion of a unified app. Potential use cases include creating an all-in-one IoT controller or a personalized news headline collection. A-Mash can also extend to AR, mapping app UIs to physical objects in AR scenes. Our prototype on Android OS shows A-Mash effectively supports UI mashups with minimal performance impact, as validated by user studies in real-world scenarios.

Being able to use a single app across multiple devices can bring novel experiences to the users in various domains including entertainment and productivity. For instance, a user of a video editing app would be able to use a smart pad as a canvas and a smartphone as a remote toolbox so that the toolbox does not occlude the canvas during editing. However, existing approaches do not properly support the single-app multi-device execution due to several limitations, including high development cost, device heterogeneity, and high performance requirement. In this paper, we introduce FLUID-XP, a novel cross-platform multi-device system that enables UIs of a single app to be executed across heterogeneous platforms, while overcoming the limitations of previous approaches. FLUID-XP provides flexible, efficient, and seamless interactions by addressing three main challenges: i) how to transparently enable a single-display app to use multiple displays, ii) how to distribute UIs across heterogeneous devices with minimal network traffic, and iii) how to optimize the UI distribution process when multiple UIs have different distribution requirements. Our experiments with a working prototype of FLUID-XP on Android confirm that FLUID-XP successfully supports a variety of unmodified real-world apps across heterogeneous platforms (Android, iOS, and Linux). We also conduct a lab study with 25 participants to demonstrate the effectiveness of FLUID-XP with real users.

The growing trend of multi-device ownerships creates a need and an opportunity to use applications across multiple devices. However, in general, the current app development and usage still remain within the single-device paradigm, falling far short of user expectations. For example, it is currently not possible for a user to dynamically partition an existing live streaming app with chatting capabilities across different devices, such that she watches her favorite broadcast on her smart TV while real-time chatting on her smartphone. In this paper, we present FLUID, a new Android-based multi-device platform that enables innovative ways of using multiple devices. FLUID aims to i) allow users to migrate or replicate individual user interfaces (UIs) of a single app on multiple devices (high flexibility), ii) require no additional development effort to support unmodified, legacy applications (ease of development), and iii) support a wide range of apps that follow the trend of using custom-made UIs (wide applicability). Previous approaches, such as screen mirroring, app migration, and customized apps utilizing multiple devices, do not satisfy those goals altogether. FLUID, on the other hand, meets the goals by carefully analyzing which UI states are necessary to correctly render UI objects, deploying only those states on different devices, supporting cross-device function calls transparently, and synchronizing the UI states of replicated UI objects across multiple devices. Our evaluation with 20 unmodified, real-world Android apps shows that FLUID can transparently support a wide range of apps and is fast enough for interactive use.

In recent years, the explosion of diverse smart devices such as mobile phones, TVs, watches, and even cars, has completely changed our lives. We communicate with friends through social network services (SNSs) whenever we want, buy stuff without visiting shops, and enjoy multimedia wherever we are, thanks to these devices. However, these smart devices cannot simply interact with each other even though they are right next to each other. For example, when you want to read a PDF stored on a smartphone on a larger TV screen, you need to do complicated work or plug in a bunch of cables. In this paper, we introduce M+, an extension of Android that supports cross-device functionality sharing in a transparent manner. As a platform-level solution, M+ enables unmodified Android applications to utilize not only application functionalities but also system functionalities across devices, as if they were to utilize them inside the same device. In addition to secure connection setup, M+ also allows performing of permission checks for remote applications in the same way as for local. Our experimental results show that M+ enables transparent cross-device sharing for various functionalities and achieves performance close to that of within-device sharing unless a large amount of data is transferred.

In this article, an adaptive real-time communication system is proposed that leverages the software-defined networking (SDN) paradigm to provide end-to-end deadline guarantees. SDN is an emerging networking paradigm that allows control of the network through directly programmable software controllers, offering the flexibility to manage and optimize network resources dynamically. By utilizing these SDN features, a novel networking system, called RT-SDN, is presented that can effectively support real-time communication through cooperative routing and scheduling. To this end, we present a highly efficient routing algorithm that can adaptively reconfigure the routes of existing flows to find bandwidth-guaranteed routes for all flows. We then introduce a new priority assignment scheme that can achieve better deadline guarantees (i.e., better schedulability). In addition, RT-SDN allows routing and scheduling to operate together by employing a feedback loop between them. We implemented a prototype of RT-SDN as an SDN controller and deployed it on a testbed network of 30 BeagleBone devices. Our experiment results show that RT-SDN can be deployed in real-world commodity networks to provide end-to-end deadline guarantees, and the proposed schemes collectively improve the schedulability to a significant degree.