Your Activities of Daily Living (YADL): An Image-based Survey Technique for Patients with Arthritis

Your Activities of Daily Living (YADL): An Image-based Survey Technique for Patients with Arthritis

Longqi Yang†,\ddagger, Diana Freed\ddagger, Alex Wu*, Judy Wu\ddagger, John P. Pollak\ddagger, Deborah Estrin†,\ddagger
Department of Computer Science, Cornell University; \ddaggerCornell Tech
*MFA Design for Social Innovation, School of Visual Art; {diana.freed, alexmtwu, judyzhaoxinwu};
{jpp9, de226}

Healthcare professionals use Activities of Daily Living (ADL) to characterize a patient’s functional status and to evaluate the effectiveness of treatment plans. ADLs are traditionally measured using standardized text-based questionnaires and the only form of personalization is in the form of question branching logic. Pervasive smartphone adoption makes it feasible to consider more frequent patient-reporting on ADLs. However, asking generic sets of questions repeatedly introduces user burden and fatigue that threatens to interfere with their utility. We introduce an approach called YADL (Your Activities of Daily Living) which uses images of ADLs and personalization to improve survey efficiency and the patient-experience. It offers several potential benefits: wider coverage of ADLs, improved engagement, and accurate capture of individual health situations. In this paper, we discuss our system design and the wide applicability of the design process for survey tools in healthcare and beyond. Interactions with with a small number of patients with Arthritis throughout the design process have been promising and we share detailed insights.

Your Activities of Daily Living (YADL): An Image-based Survey Technique for Patients with Arthritis

Longqi Yang†,\ddagger, Diana Freed\ddagger, Alex Wu*, Judy Wu\ddagger, John P. Pollak\ddagger, Deborah Estrin†,\ddagger
Department of Computer Science, Cornell University; \ddaggerCornell Tech
*MFA Design for Social Innovation, School of Visual Art; {diana.freed, alexmtwu, judyzhaoxinwu};

{jpp9, de226}



\begin{picture}(20.0,6.0)\put(0.0,-0.95){\crnotice{\parbox[b]{240.0pt}{}}} \end{picture}\end@float

Categories and Subject Descriptors H.4 [Information Systems Applications]: Miscellaneous; H.5.2 [Information Interfaces and Presentation]: User Interfaces

  • Arthritis; Activities of Daily Living; Survey; Self-Report; Health; Pain

    Measuring and understanding an individual’s functional state is key to effective treatment of patients with chronic conditions[?, ?]. In this paper, we focus on Patients with Arthritis (PwA), including Rheumatoid Arthritis (RA) and Osteoarthritis (OA). The symptoms and health conditions of PwA are largely manifested in their performances of Activities of Daily Living (ADL) [?], due to muscle weakness, inflammation and limitations caused by joint pain and stiffness [?, ?]. Timely active recordings of the difficulties in performing ADLs by patients informs clinicians of treatment effectiveness and clinical changes [?]. Careful analysis of ADL functional trends is also helpful in prioritizing and scheduling rehabilitative intervention [?]. As the most widely adopted method of collecting information with respect to ADL functionality [?], a variety of short and long-form questionnaires have been designed and evaluated in both inpatient and outpatient settings [?, ?]. While the answers to questions have been validated as correlated and effective to varying degrees [?], our formative interviews identified three inherent limitations to these static forms:

    Coverage: Although activities included in a single standardized questionnaire have been carefully selected to be clinically relevant, it is not possible for these to capture all ADLs that might be difficult for all patients. For instance, in the Boston AM-PAC form [?], walking, running and bending are included as they are common and basic measures of mobility, but there are no questions pertaining to turning a doorknob or typing on a keyboard. For many patients, these omissions would result in failure to capture challenges related to joint stiffness and an overall under-representation of ADL difficulty. This concern arose in our patient interviews and will be discussed in the following sections.

    Legibility: Some items presented in questionnaires are designed specifically to capture patients’ difficulties with fine-grained ADLs but may not be relevant to an individual patient. For instance, the following activity descriptions in the standard Boston AM-PAC form, “Bending over from a standing position to pick up a piece of clothing from the floor without holding onto anything” and “Standing up from a low, soft couch”, are sometimes too detailed or not relevant to the individual patient experiences. This impacts both the user experience and effectiveness of the survey.

    Availability: Standard ADL assessments are typically only completed before, during or after a clinical encounter in an office setting. For example, the Health Assessment Questionnaire (HAQ) [?] is administered when the patient registers for a clinical visit [?]. As a result, the patient’s day to day ADLs and variability in performance are usually undersampled [?]. Additionally, data is subject to considerable bias as answers are most often based on patients’ retrospective recall [?].

    We present our system named YADL (Your Activities of Daily Living) - an image-based survey inspired by Photographical Affect Meter (PAM) [?], designed to improve both fidelity and user experience in reporting ADLs. Compared with traditional questionnaires, YADL is designed to be personalized, engaging and easily accessible. (1) Personalized experiences: Instead of comprehensively listing all possible ADLs via word-based descriptions, we leverage images’ inherent ambiguities that allow for personalized interpretation and thus capture fine-grained ADL experiences; moreover, the inventory of images can be further tailored to specific demographics and disease contexts. (2) Patient engagement: We use representative activity images that are quick and easy to interpret. We argue that these photos are easier to understand and more effective at capturing ADL’s data as well. (3) Wide availability: YADL can be accessed via a web browser on desktop computers or on mobile devices so that patients can complete YADL at their convenience to keep timely records of their functional states. Photos also provide greater accessibility for lower literacy and non-english-speaking patients.

    We have designed YADL using a patient-driven participatory design process to iteratively explore and obtain feedback from clinicians and arthritis patients. Preliminary user testing suggests that patients’ responses on YADL exhibit consistency with two widely-used instruments, i.e. Boston AM-PAC [?] and WOMAC [?]. In addition, the feedback collected during the design process uncovers several added benefits from YADL that could provide unprecedented information in understanding patients’ health conditions.

    In this paper, we present the detailed design approach and discuss the lessons learned throughout the process of user interviews, rapid prototyping, user testing and iterative refinement. We further propose that this exploration of image-based applications as a replacement for short and long-form text questionnaires will inspire new work in healthcare and beyond. Finally, the iterative design process that we adopt in this paper can be widely applied to the general task of standard survey improvements and the transition from text-dominated interfaces to visually-rich interfaces. Future work, informed by the detailed feedback in the preliminary user testing, will be conducted to validate these results with a larger randomized sample.

    YADL is a responsive web application designed to be accessible across a wide range of devices. At the beginning, the patient is presented with an activity image and asked to indicate “How difficult is this activity for you on a difficult day?” The patient then selects his/her response from the choices of Easy, Moderate and Hard, or Skip if the image is irrelevant or hard to interpret. After making a selection, another image is immediately presented. This continues until all images in the database have been evaluated. An example of user interface is shown in Fig.1(a). Patients with arthritis may have accessibility issues due to the stiffness and/or pain in the fingers. Our design addresses these concerns by using color-coded large buttons and a simple one tap gesture.

    Figure \thefigure: YADL user interface on mobile web browser

    To identify and explore the design opportunities, we conducted both telephone and in person interviews with three arthritis patients to understand their previous experiences with traditional ADL questionnaires.

    Legibility. When we asked the question “Did you feel that the traditional questionnaire captured your health functionality?” in the interview, two patients responded, “No… I rush through paper forms at MD…” (P1) and “No, I go through as fast as I can…” (P2), suggesting that the form-based approach to ADLs is ineffective in engaging patients and insufficient for capturing functional states of ADLs. Significant opportunity exists to improve on the patient experience and hence reporting fidelity.

    Availability. The granularity of standard assessments is greatly affected by the frequency of clinician encounters, We asked the patients “How often do you see your doctor or another healthcare professional for your RA/OA/PA?”. Participants maintained office visits at “6 month” (P1), “once a month” (P2) and “A couple of times a year” (P3). As a result, patients’ everyday or even week to week variations of ADL functionality are not captured and communicated in the current system. Retrospective recall, at the time of the clinical encounter is subject to recall bias. With the mobile first principle in mind, the survey can be easily accessed and completed via mobile devices and provide opportunities for fine-grain data capture.

    Coverage. When we asked the patients to describe how pain and flare-ups impact their related daily life, responses included, “I cannot twist a cap and open certain doors… I am afraid of getting hit by a bike so I take a long walk everyday and only walk in the same neighborhood” (P3), “Stepping into a bathtub is dangerous” (P2). The activities mentioned in the responses, e.g. open doorknobs, twist a cap and step into bathtub, are not reflected in traditional questionnaires, such as Boston AM-PAC and WOMAC, so that they are insufficient to capture patients’ actual health conditions. Therefore, survey instruments that cover wider range of ADLs are needed to alleviate this limitation.

    Bathing, Feeding, Dressing independently, Walking, Climbing stairs, Rising from chair, Yard work, Driving, Errands, Cooking, Feeds self, Holds drinking cup, Washing, Lifting groceries, Hair brushing, Buttoning shirt, Zipping coat, Exercise, Work, Cooking, Kneeling, Bending , Stooping, Gardening, Shaving, Brushing teeth, Open doors, Faucet handles, Door knobs, Hold pens, Hold silverware, Pick up glass, Open jar, Mopping, Sweeping, Vacuuming, Cutting food, Squeezing toothpaste, Blowdrying hair, Open carton of milk, Socializing, Using scissors, Parties, Theatre, Dining out
    Table \thetable: Identified activity list

    Activity images are the key component of our application. We started by identifying key activities through a two-part process. First, we performed a comprehensive literature search to determine the most commonly used ADL assessment forms and extracted the activity inventory from each. We spoke with rheumatologists and occupational therapists and conducted a retrospective review of ADL assessments as applied to the arthritis population. The activity phrases were extracted from the main verbs in the articles, e.g. standing up, walking, running etc., such that they are general enough for patients to have personalized interpretations. Second, we identified online forums and communities as important venues for understanding real situations of patients and the hard ADLs that are common among the population. We gathered data from support and wellness communities, e.g. inspire; community forums on patient advocacy organizations, e.g.; social networks, e.g. MyRATeam; peer-to-peer support forums, e.g. patientslikeme and rheum4us; and news websites, e.g., where patients regularly shared their health conditions and day-to-day personal stories. From their descriptions, we extracted activity phrases pertaining to everyday ADLs, i.e. common verbs that were mentioned in the online posts, and use them as additions to traditional questionnaires. With an aggregated list of ADLs based on common assessments and online posts, we then worked with clinicians and patients to review and further refined the list. The final list of identified key items are presented in Table.Your Activities of Daily Living (YADL): An Image-based Survey Technique for Patients with Arthritis.

    For each activity phrase, we searched on Flickr and Google for images with Creative Commons Licenses for our initial images which we reviewed with a clinician and 3 patients. However, through the interviews, we found that stock images alone did not include appropriate details and enough demographic diversity. For example, we presented patients with images of a lever doorknob and learned that difficulty is consistent with round doorknobs (P1, P2). Similarly, we showed individuals walking on a paved surface and through user interviews, we learned the importance of allowing patients to customize images to represent gravel, snow, grass, dirt and sand as different surfaces were relevant for different individual patients (P1, P2).

    To complement the limitations of the public images, we then worked with a photographer and recruited volunteers to pose for photo shoots in various environmental settings, e.g. walking on different surfaces. All of the images were cropped to focus on the particular activities and none of them include the subjects’ faces. We further improved the image library by getting feedback on several variations from both clinicians and patients. Each matched photo was shot several times, in different settings, to assure that the images we captured and tested can be understood quickly. As a result, 47 activity images were filtered for YADL. A grid of sample images is shown in Fig.2. Through the patient interviews, we further discovered that the images included possess two added benefits in understanding patients’ health conditions, which are discussed in detail next:

    Figure \thefigure: Sample activity images used in YADL (Indexes below)

    Encourage personal interpretations: Specific images are inherently ambiguous for some individuals due to cultural and characteristic differences. This phenomenon is shown in the data that we collected. However, we observe that this type of image also provided opportunities for personal interpretations and thus helped to explore hard ADLs that can not be covered by simply listing them with text descriptions. For example, regarding image I-19, although P3 felt unclear about the content, “Watching the horizon? Not clear…”, P1 and P2 successfully interpreted it and P2 even brought up her related personal story. “Socializing” (P1), “Sitting in a restaurant…I can’t do it with neck and back pain.” (P2). This also happened for image I-28. While P3 felt it to be unclear, “Picture is not clear. Are you asking if I turn on the water?…” , P2 made strong statement of how this activity will affect her life, saying “This is a bathtub…I can’t bend my body to get into a bath.” These personalized experiences are hard to capture with static questionnaires and we believe that visual contents could play an important role in engaging and understanding PwA.

    Catalyze story sharing: The activity images could also generate conversations between the patients and the their care providers about their pain-related personal stories, which are helpful in understanding the changes in their health conditions and impact on their day-to-day lives. For example, the image of people socializing (I-19) generated a discussion about the impact of RA on social life and how “flare-ups force me to stay home and made me depressed.”(P2) Images of door handles (I-32) reminded two patients (P1, P2) of specific use cases of lever doorknobs vs. twist doorknobs, which highlights individual patient concerns about recent changes regarding management of daily care. Typically, within limited clinical office visit time it is difficult to hit upon the right questions for each patient to evaluate functionality and go through detailed dialogs, the illustrative experiences that brought by images could be an useful addition for physician assistants, nurse practitioners, occupational and physical therapists, care coordinators and family members to understand patients’ individual situations.

    Our design process included iterative testing with three patients diagnosed with different types of arthritis (P1:OA, P2:RA, P3:OA). Participants were recruited from Facebook patient communities. Testing consisted of three components: (1) Each participant was asked to assess him/herself using all 47 images in our exploratory inventory for YADL and his/her responses were recorded. (2) Each patient was granted access to a web-based version of the Boston AM-PAC and WOMAC forms and instructed to fill out both forms respectively. (3) A telephone interview followed during which the participants rated the experience and gave feedback. By comparing their choices on different types of instruments and analyzing their conversations with the interviewers, we were able to compare the functionality of the YADL survey to that of standard instruments. The feedback provided by the participants suggests that YADL could be an effective and promising tool for capturing the fine-grained functional state of ADLs for patients with arthritis.

    Comparison of YADL to standard assessments. We compared the results of YADL with those from gold-standard ADL surveys and our preliminary results suggest that YADL can be easily understood by the patients and that their selections accurately reflect the functional states of ADLs. In our testing, we chose the Boston AM-PAC form and WOMAC as the assessments. To make our system and text-based questionnaires directly comparable, we quantified the available choices in YADL, i.e. Easy = 0, Moderate = 1, Hard = 2. Also, each question in the standard form was pre-matched with a set of related activity images (i.e. sharing the same verb). As a result, 18/47 and 25/47 images were necessary and sufficient to match all questions in the AM-PAC form and WOMAC respectively. The metrics that we compare are the highest response value on the image set and the patient’s choice on the questionnaire. While the number of patients is too small for real inference, Pearson correlations are 0.743 and 0.747 between patient responses to YADL and AM-PAC and WOMAC suggesting that the photographical interface on YADL can be effective and accurate in capturing difficulties of ADLs for PwA.

    Covering a wider range of ADLs. Compared with standard assessments such as Boston AM-PAC and WOMAC, YADL captures wider range of ADLs that are difficult for the patients. In the question-image matching process mentioned above, 29/47 and 22/47 images remained uncovered by AM-PAC and WOMAC respectively. Among all these unmatched activity photos, 15/29 and 11/22 of them are expressed as hard ADLs by at least one of the participants. For instance, P2 clicked hard for image I-12 (Fig. 2), typing on keyboard, image I-19, sitting in the restaurant (socializing), and image I-14, walking on the treadmill, which are missing in both forms. The above results justify the value of the comprehensive literature search and the activity phrases extraction incorporated into YADL design as this process ensures wider coverage of ADLs with our system.

    Overall patient evaluation. In our testing among three patients, YADL was favored by two of the participants over standard questionnaires. For the question “Which method would do you think best captures how you are doing on a day to day basis? (Questionnaire/Neither/YADL)”, P1 and P2 expressed strong preferences for YADL. “… liked pictures with people in them…” (P1), “Liked image much more … somewhat I like the one with pictures, just easier to see it and feel it.” (P2). Although P3 responded neutrally to this question, “either could work. I am used to the forms and not used to the pictures but they are fine.”, we did observe that YADL engaged her with finer-grained ADL captured.

    The iterative process that we present in this paper is applicable to other domains of human computer interaction where current text-dominated surveys and interfaces would benefit from this visually-rich approach:

    Key item identification from broad assessments and online articles. With the overwhelming amount of information provided by variable standard questionnaires, the process of aggregating and harvesting key items would benefit the design in terms of wide coverage in the initial prototype. In addition, online articles are important sources of information about individual situations. Incorporating this information could potentially provide unprecedented utilities for end users. This process could also be used to refine existing forms as the information from patients’ daily posts could provide directions for the changes.

    Fast iterative refinement with inputs from field professions. It’s critical to carry the iterative design fast along with field professions. This is the process in which survey items are adjusted and refined quickly based on the inputs from domain experts. In the design of text-to-visual interfaces, domain knowledge is needed for the conversion so as to guarantee that the alternative representations are understandable and scientifically valid.

    Personalized activity images that patients indicate as hard in the survey can further fuel a customizable self-report tool that enables timely and quick-tap records of their functional states, as Fig. 1(b) shows. The tool could base on the reduced custom set of images that patient has chosen and thus provide finer-grained data points for clinicians to understand patients’ health conditions.

    In this paper, we introduced the design of YADL, an image-based survey technique for patients with arthritis. The iterative design process presented in this paper: identify key items from broader resources and regularly elicit inputs from field professions, can be adapted to transform text-dominated interfaces to visually-rich interfaces more generally in healthcare and beyond. Next steps for YADL will be the validation of the system at scale and development the self report tool. We will scale the number of participants and test it in a longitudinal study to validate its value in understanding ADL and patient function.

    • [1] B. Archenholtz and B. Dellhag. Validity and reliability of the instrument performance and satisfaction in activities of daily living (ps-adl) and its clinical applicability to adults with rheumatoid arthritis. Scandinavian Journal of Occupational Therapy, 15(1):13–22, 2008.
    • [2] B. Bruce and J. F. Fries. The stanford health assessment questionnaire: a review of its history, issues, progress, and documentation. The Journal of Rheumatology, 30(1):167–178, 2003.
    • [3] C. A. Ciro, M. P. Anderson, L. A. Hershey, C. I. Prodan, and M. B. Holm. Instrumental activities of daily living performance and role satisfaction in people with and without mild cognitive impairment: A pilot project. American Journal of Occupational Therapy, 69(3):6903270020p1–6903270020p10, 2015.
    • [4] S. M. Haley, P. L. Andres, W. J. Coster, M. Kosinski, P. Ni, and A. M. Jette. Short-form activity measure for post-acute care. Archives of Physical Medicine and Rehabilitation, 85(4):649–660, 2004.
    • [5] P. P. Katz. The impact of rheumatoid arthritis on life activities. Arthritis & Rheumatism, 8(4):272–278, 1995.
    • [6] S. Katz. Assessing self-maintenance: activities of daily living, mobility, and instrumental activities of daily living. Journal of the American Geriatrics Society, 31(12):721–727, 1983.
    • [7] B. E. M. Lawton M. Powell. Assessment of older people: self-maintaining and instrumental activities of daily living. 1969.
    • [8] L. Legg, A. Drummond, and P. Langhorne. Occupational therapy for patients with problems in activities of daily living after stroke. The Cochrane Library, 2006.
    • [9] C. A. Mancuso and M. E. Charlson. Does recollection error threaten the validity of cross-sectional studies of effectiveness? Medical Care, pages AS77–AS88, 1995.
    • [10] N. Matsumura, H. Nishijima, S. Kojima, F. Hashimoto, M. Minami, and H. Yasuda. Determination of anaerobic threshold for assessment of functional state in patients with chronic heart failure. Circulation, 68(2):360–367, 1983.
    • [11] S. McConnell, P. Kolopack, and A. M. Davis. The western ontario and mcmaster universities osteoarthritis index (womac): a review of its utility and measurement properties. Arthritis care & research, 45(5):453–461, 2001.
    • [12] T. Pincus, T. Sokka, and A. Kavanaugh. Quantitative documentation of benefit/risk of new therapies for rheumatoid arthritis: patient questionnaires as an optimal measure in standard care. Clin Exp Rheumatol, 22(5 Suppl 35):S26–33, 2004.
    • [13] J. P. Pollak, P. Adams, and G. Gay. Pam: a photographic affect meter for frequent, in situ measurement of affect. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 725–734. ACM, 2011.
    • [14] P. PROMIS. Patient reported outcomes measurement information system. National Health Institute, 2012.
    • [15] A. L. Stewart, S. Greenfield, R. D. Hays, K. Wells, W. H. Rogers, S. D. Berry, E. A. McGlynn, and J. E. Ware. Functional status and well-being of patients with chronic conditions: results from the medical outcomes study. Jama, 262(7):907–913, 1989.
    • [16] A. A. Stone and S. Shiffman. Capturing momentary, self-report data: A proposal for reporting guidelines. Annals of Behavioral Medicine, 24(3):236–243, 2002.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description