While many devices today increasingly have the ability to predict human activities, it is still difficult to build accurate personalized machine learning models. As users today will become responsible for helping to train their own models, we are interested in ways for applications to request labeled data from their users in a non-invasive way. This work focuses on opportunities for intelligent systems to ask their users for help through interactions over an extended period of time in order to improve their machine learning models. We focus on trading off the expected increase in accuracy with the potential interruptions that the questions may cause to improve the usability of such systems. Keywords experience sampling, asking for help, machine learning, long-term interactions ACM Classification Keywords H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. I.2.m Artificial Intelligence Miscellaneous General Terms Algorithms, Human Factors Research Situation I am a...