Cohort

A cohort groups users who keep track of their food consumption. It is identified by a unique name and managed by cohort collaborators.

A cohort is associated with food lists, meaning any dish in this cohort must be annotated using only those food lists’ foods.

User

A user is a person who tracks their food consumption through the MyFoodRepo mobile app. A user can participate (via participations) in only one cohort at a time.

Participation

A participation links a user to a cohort. Typically, a cohort manager (collaborator) creates a participation for that cohort and shares the key with a user. The user can then use that key to join the cohort by entering it in the mobile app.

When done, this sets the associated at datetime attribute of the participation object. The started at and ended at datetime attributes can be set to specify the duration of the participation. If the ended at datetime attribute is set and is in the past, the user can no longer upload pictures to the cohort.

As MyFoodRepo doesn’t store participant emails, the cohort manager must keep a list of participation token/user email associations. If a user changes devices or logs out, you’ll need to reset the participation token to allow them to sign in again to the MyFoodRepo mobile app.

Collaborator

A collaborator manages cohorts of users through the web app. They are identified by a name and email address, and can be associated with multiple cohorts through collaborations objects.

Each collaborator has a sitewide role that controls system-wide access:

  • basic: Standard collaborator access (default).
  • developer: Access to developer-only sections (e.g., diagnostics).
  • admin: Full administrative access (organizations, all cohorts, all collaborators).

In addition, each collaboration carries a cohort-level role that controls what the collaborator can do in that specific cohort:

  • manager: Manages the cohort (participations, food lists, exports, etc.).
  • annotator: Annotates the dishes captured by the users of the cohort.

Collaboration

A collaboration links a collaborator to a cohort. A collaboration starts on the started_at datetime and ends on the ended_at datetime attribute.

Dish

When logged into the mobile app, a user can capture their meal consumption by either taking a picture of a dish, scanning a product barcode, or entering a text description of the dish.

The mobile app sends this information to the web backend through the API.

The web backend then creates a dish object associated with the user who took the picture and with an annotation, which itself is associated with an intake capturing the time and “location” (timezone) of the dish consumption.

When the user re-logs the same meal later via the “Quick pick” feature of the mobile app, a new annotation is created (with its own intake) and linked to the original through parent_annotation_id. Each annotation therefore corresponds to a single meal event.

Intake

An intake captures a single consumption event of a dish: the date and time the user ate the meal (consumed_at) and the timezone of the device at that moment.

In practice, each annotation has exactly one intake. When a user re-logs the same meal via the “Quick pick” feature, the mobile app does not add a new intake to the original annotation — it creates a new child annotation (linked via parent_annotation_id) with its own intake and a fresh copy of the annotation items.

The consumed_at of an intake must fall within the participation’s started_at/ended_at window. Intakes feed the daily nutrient and HEI computations: whenever an intake is created, updated, or destroyed, the affected day(s) are scheduled for refresh.

Segmentation

When the dish is created with a picture, it is also associated with a segmentation object. The segmentation sends the picture to an automated segmentation service that creates the annotation items for each food detected.

Annotation

The annotation is a key object in the system as it links the dish, its intakes, a cohort, and the foods or products present in the dish.

It is associated with multiple annotation item objects that capture which foods or barcoded products are present in the dish. Annotation items are created by an automated segmentation service or by a human annotator (collaborator).

It can also be associated with multiple comment objects. The comments are used to communicate with the user and ask for information about the dish.

Lifecycle

A typical annotation goes through the following lifecycle:

stateDiagram-v2
[*] --> initial
annotated --> [*]
initial --> awaiting_segmentation_service : annotation sent to segmentation service
awaiting_segmentation_service --> annotatable : segmentation service performed task
annotatable --> info_asked : annotator asks question
annotatable --> annotated : annotator finishes annotation
info_asked --> requires_human_intervention : user answers annotator's question
annotatable --> awaiting_llm_service : user posts comment
awaiting_llm_service --> annotatable : chatbot answers
awaiting_segmentation_service --> requires_human_intervention : segmentation service fails
awaiting_llm_service --> requires_human_intervention : chatbot cannot answer
requires_human_intervention --> annotated : annotator finishes annotation
info_asked --> annotated : annotator finishes annotation
annotated --> annotatable : annotator clicks "Allow annotation"
  • initial: The state of a newly created annotation, before the segmentation service has been called.
  • awaiting_segmentation_service: The state of an annotation after the segmentation service has been called. The segmentation service is an external service that takes a dish picture and returns a list of annotation_items objects that capture the location of the food in the dish picture. The segmentation service is called asynchronously, so the annotation is in this state until the segmentation service has finished.
  • annotatable: The state of an annotation after the segmentation service has finished. The annotation is now ready to be annotated by a human annotator, or commented on by a user (and answered by the chatbot).
  • info_asked: The state of an annotation after the human annotator has asked the user for more information about the dish. The user can then answer the question by adding a comment to the annotation.
  • awaiting_llm_service: The state of an annotation after a user posted a comment and is waiting for the chatbot to answer.
  • requires_human_intervention: The state of an annotation after a user answered an annotator’s comment, or when the user sends too many comments and the chatbot fails to make the appropriate changes to the dish, or when the segmentation service fails to analyze the image.
  • annotated: The state of an annotation after the human annotator has finished annotating the dish.

Once the annotation is in the annotated state, updates and commenting are disabled. To re-enable them, the annotation must be reset to the annotatable state by clicking on the “Allow annotation” button on the annotation page.

Annotation item

An annotation item links a dish to a food or product. It is associated with a PolygonSet object that captures the location of the food or product in the dish picture. It is also associated with an Item object that captures the food or product information.

Comment

A comment is a message attached to an annotation, used as the conversation thread between the user, the human annotator, and the chatbot. Every comment has exactly one author, which is either:

  • a user (typed in the mobile app)
  • a collaborator (typed in the annotation interface)
  • the chatbot (bot flag set), or a system message (welcome and kcal_verification are dedicated flags for the welcome message and the calorie-verification request)

A comment can be marked silent, in which case the recipient does not receive a push notification — typically used by annotators for acknowledgements that don’t require a response from the user.

Posting a comment also influences the annotation lifecycle: a user comment on an annotatable annotation hands it off to the chatbot (awaiting_llm_service), and a user’s answer to an annotator’s question moves it to requires_human_intervention.

Food list

A food list groups foods from the same source and version. It can be editable or not. Typically, a food list imported from an official source is not editable, while a custom food list created manually for a specific cohort is editable.

Food

A food is associated with a specific food list and might be editable depending on the food list it belongs to.

Product

MyFoodRepo caches product data in its own database. Product data comes from FoodRepo and OpenFoodFacts. When a user scans a product barcode and the product does not already exist in the MyFoodRepo database, the user will be asked to take at least five pictures of the product, which are recorded together with the barcode in the database. MyFoodRepo backend will then check on FoodRepo and OpenFoodFacts if data already exists for this product. If it does, the data is imported into the MyFoodRepo database. If it does not, the product pictures will be uploaded to FoodRepo for parsing. The MyFoodRepo backend will then periodically check for updated product data.