One of the important components of the lifelog systems is the user interface which provides the ability to quickly and easily find a specific image or set of images. Although lifelogging is a mature field in the information retrieval domain, the focus on user interfaces is not explored extensively. We start by identifying the common issues with existing lifelog systems from the user interface and user experience perspective. Following the exploration, we present a set of guidelines for designing a user interface for Lifelog systems. We introduce LifeLens- a novel minimalist user interface design specifically designed to improve the usability and ease of use of an interactive lifelog system. The initial version of the LifeLens system provides several improvements over existing lifelog systems addressing the design issues identified during the exploration. The proposed system presents several features that not only enable the users of the system to easily navigate the interface with minimal effort on the user’s part to learn and understand the features offered but also provide a minimal way to gather user feedback.

In this paper, we introduce Voxento 4.0 – an interactive voice-based retrieval system for lifelogs which has been developed to participate in the sixth Lifelog Search Challenge LSC’23, at ACM ICMR’23. Voxento has participated three times in the LSC editions and achieved the rank of 4th in LSC21 and 5th in LSC22 respectively. In this version, Voxento 4.0, we have focused on improving the previous system’s interface, voice interaction and retrieval functionality. The current version has implemented some processing and cleaning of the dataset and employs the CLIP model to extract image features. In addition, the system’s interface was redesigned for better visualisation of the elements and the images for effective interaction. This improvement in the interface will help to support voice interaction in future work. The interface developments include logging voice interaction and images displayed, submitted, selected and starred to enhance user experience with the system. The voice interaction part has also been enhanced in the workflow of the voice lifecycle interaction and with additional voice commands.

Lifelogging is referred to as the process of automatically capturing the everyday activities of an individual to ultimately create a digital diary for further sharing, which can be challenging to manage and retrieve due to its multimodal nature. Lifelog retrieval systems not only have the potential to transform the way people interact and understand their lives, but also provide insights into their behaviour, habits, and preferences. The Lifelog Search Challenge (LSC) is a live benchmarking challenge to evaluate the performance of lifelog search tools in a real-time. This paper describes the modifications made to the E-LifeSeeker retrieval system, which participates in the 6th LSC challenge. This year, we enhance not only the underlying core engine with the latest pre-trained embedding models but also the user interface to be more intuitive for novice users. Moreover, Differential Networks are implemented to address the new question-answering task this year. These new modalities are designed to provide users with a more intuitive and efficient search experience, easing the process of locating information needed from the huge collection of egocentric lifelog images.

The continuous collection and storage of personal data, denoted Lifelogging, has gained popularity in recent years as a means of monitoring and improving personal health. One important aspect of lifelogging is the collection and analysis of image data, which can provide valuable insights into an individual’s lifestyle, dietary habits, and physical activity. The Lifelog Search Challenge provides a unique opportunity to explore the state-of-the-art in lifelogging research, particularly in the area of egocentric image retrieval and analysis. Researchers can propose their approaches and compete to solve lifelog retrieval challenges and evaluate the effectiveness of their systems on a rich multimodal dataset generated by an active lifelogger with 18 months of continuous capture of lifelogging data. This paper presents the second version of MEMORIA, a computational tool developed to participate in the Lifelog Search Challenge 2023. In this new version, the information retrieval is based on the use of natural language search with the possibility to filter the results based on keywords and time periods. The system applies image analysis algorithms to process visual lifelogs, from pre-processing algorithms to feature extraction methods, in order to enrich the annotation of the lifelogs. This new version explores the use of a graph database, more detailed image annotation, and event segmentation, in order to improve the performance and user interaction. Experimental results of the user interaction with our retrieval module are presented, confirming the effectiveness of the proposed approach and showing the most relevant functionalities of the system.

Retrieval is a fundamental challenge within the research community of lifelog and the Lifelog Search Challenge (LSC) has been an important annual benchmarking activity for interactive lifelog retrieval systems since 2018. This paper proposes MyEachtra (/mai-AK-truh/), a system designed for the upcoming LSC’23 workshop. Improved upon MyScéal, which was the top performing system from LSC’20 to LSC’22, MyEachtra includes modifications to address the challenges of non-owner user understanding of lifelog contexts and open-ended lifelog question answering. Specifically, MyEachtra shifts the focus from images to events as retrieval units. Events are segmented using location metadata as well as visual and time differences between successive images. A pilot study on different approaches to aggregate images into events was conducted to test the automatic performance of the system, which showed promising results. For known-item queries, showing only the top 3 events proved to be adequate to find relevant images. However, future evaluation of the performance for ad-hoc and question-answering queries is necessary for a complete analysis of the MyEachtra.

Lifelogging is an activity of recording all events that happen in the daily life of an individual. The events can contain images, audio, health index, etc which are collected through various devices such as wearable cameras, smartwatches, and other digital services. Exploiting lifelog data can bring significant benefits for lifeloggers from creating personalized healthcare plans to retrieving events in the past. In recent years, there has been a growing development of interactive lifelog retrieval systems, such as competitors at the annual Lifelog Search Challenge (LSC), to assist lifeloggers in finding events from the past. This paper introduces an interactive lifelog image retrieval called MemoriEase for the LSC’23 challenge. This system combines concept-based and embedding-based retrieval approaches to answer accurate images for LSC’23 queries. This system uses BLIP for the embedding-based retrieval approach to reduce the semantic gap between images and text queries. The concept-based retrieval approach uses full-text search in Elasticsearch to retrieve images having visual concepts similar to keywords in the query. Regarding the user interface, we make it as simple as possible to make novices users can use it with only a small effort. This is the first version of MemoriEase and we expect this can help users perform well in the LSC’23 competition.

As part of the 6th Lifelog Search Challenge, this paper presents an approach to arrange Lifelog data in a multi-modal knowledge graph based on cluster hierarchies. We use multiple sequence clustering approaches to address the multi-modal nature of Lifelogs in relation to temporal, spatial, and visual factors. The resulting clusters, along with semantic metadata captions and augmentations based on OpenCLIP, provide for the semantic structure of a graph including all Lifelogs as entries. Textual queries on this hierarchical graph can be expressed to retrieve individual Lifelogs, as well as clusters of Lifelogs.

In this work, we present our system Memento 3.0 for participation in the Lifelog Search Challenge 2023, which is a successor to the previous 2 iterations of our system called Memento 1.0 [1] and Memento 2.0 [2]. Memento 3.0 employs image-text embeddings derived from OpenAI CLIP models as well as larger OpenCLIP models trained on ∼ 5x more data. Our system also significantly reduces the query processing time by almost 75% when compared to its predecessor systems by employing a cluster-based search technique. We additionally make important updates to the system’s user interface to offer more flexibility to the user and at the same time be better suited to efficiently handle new query types introduced in the Lifelog Search Challenge.

AI-assisted tools have become more prevalent than ever in the last few years. However, applying them to build a lifelog retrieval system is still non-trivial due to the disparity in interfaces and interactions. The Lifelog Search Challenge (LSC) aims to provide a testing ground where systems can be benchmarked in a highly competitive setting. In this paper, we present the fourth iteration of our participating system FIRST. For this year, we adopt generative models to equip the system with predictive ability rather than entirely relying on the user to input the query. We also index a sequence of images as an event for improved search speed. Finally, we demonstrate how the additional features can assist users in searching.

Searching substantial data archives of lifeloggers is a challenging task. The Lifelog Search Challenge (LSC) is an annually held competition with the aim of encouraging international teams to develop interactive content retrieval systems capable of searching large lifelog databases. LSC takes place as a live event co-located with the ACM International Conference on Multimedia Retrieval (ICMR), where teams compete against each other by solving retrieval tasks issued by the lifelogger. This paper presents our newest version of lifeXplore, a lifelog retrieval system that has been participating in LSC since 2018. For this year, we significantly redesign the entire system (backend, middleware, and frontend) and integrate free text-search using embeddings from vision transformers trained with large sets of text-image pairs. We present a novel architecture for multi-source search, where results from image embeddings are used together with results from traditional content analysis (for objects, concepts, and recognized text). We also perform intensive analysis of vision transformer models in order to know which one fits best to the requirements of the LSC.

In this paper, we introduce LifeInsight – an interactive lifelog retrieval system developed for the sixth annual Lifelog Search Challenge (LSC’23). LifeInsight incorporates semantic search mechanisms from state-of-the-art lifelog retrieval systems while focusing on providing insights into the lifelogger’s routine using spatial information to support question-answering tasks. The system employs the Bootstrapping Language-Image Pre-training (BLIP) model for zero-shot image-text retrieval, which has been shown to achieve higher recall scores than the CLIP model on the Flickr30K dataset. In addition, the Elastic Search filtering mechanism is utilized to remove irrelevant images. Apart from semantic search mechanisms, the system also supports visual similarity search by comparing the inner product distance between the vectors in the lifelog image corpus and the query image. Furthermore, the system includes an explicit relevance feedback function, AI-based query description rewriting, and visual-example-generating features to re-phrase the query to describe it better and support end-users envisioning the targeted image for retrieval.

Personal lifelog data collections are becoming more common as a memory aid, as well as for analytical tasks, such as health and fitness analysis. Due to the multimodal and personal nature of lifelog data, interactive multimedia retrieval approaches are required to facilitate flexible and iterative query formulation and result exploration for retrieval and analysis. In recent years, novel user interface modalities have emerged, that allow new ways for users to interact with a retrieval system. Virtual reality, one such new modality, provides advantages as well as challenges for interactive multimedia retrieval in comparison to conventional desktop-based interfaces.

This paper describes a novel desktop-virtual reality hybrid system participating in the Lifelog Search Challenge 2023. The system, which is based on the components of the vitrivr stack, is described with a focus on query formulation in the web-based desktop user interface vitrivr-ng, and result exploration in the virtual reality-based vitrivr-VR.