Cohort

A cohort groups users who keep track of their food consumption. It is identified by a name and managed by cohort collaborators.

A cohort is associated with food lists, meaning any dish in this cohort must be annotated using only those food lists’ foods.

User

A user is a person who tracks their food consumption through the MyFoodRepo mobile app. A user can participate (via participations) in only one cohort at a time.

Participation

A participation links a user to a cohort. Typically, a cohort manager (collaborator) creates a participation for that cohort and shares the key with a user. The user can then use that key to join the cohort by entering it in the mobile app.

When done, this sets the associated at datetime attribute of the participation object. The started at and ended at datetime attributes can be set to specify the duration of the participation. If the ended at datetime attribute is set and is in the past, the user can no longer upload pictures to the cohort.

As MyFoodRepo doesn’t store participant emails, the cohort manager must keep a list of participation token/user email associations. If a user changes devices or logs out, you’ll need to reset the participation token to allow them to sign in again to the MyFoodRepo mobile app.

Collaborator

A collaborator manages cohorts of users through the web app. They are identified by a name and email address, and can be associated with multiple cohorts through collaborations objects.

Each collaborator has a sitewide role that controls system-wide access:

  • basic: Standard collaborator access (default).
  • developer: Access to developer-only sections (e.g., diagnostics).
  • admin: Full administrative access (organizations, all cohorts, all collaborators).

In addition, each collaboration carries a cohort-level role that controls what the collaborator can do in that specific cohort:

  • manager: Manages the cohort (participations, food lists, exports, etc.).
  • annotator: Annotates the dishes captured by the users of the cohort.

Collaboration

A collaboration links a collaborator to a cohort. A collaborator can hold only one collaboration per cohort. A collaboration is active from the moment it is created until it is deactivated, which sets its deactivated_at datetime attribute; clearing that attribute reactivates it.

Dish

When logged into the mobile app, a user can capture their meal consumption by either taking a picture of a dish, scanning a product barcode, or entering a text description of the dish.

The mobile app sends this information to the web backend through the API.

The web backend then creates a dish object associated with the user who took the picture and with an annotation, which itself is associated with an intake capturing the time and “location” (timezone) of the dish consumption.

When the user re-logs the same meal later via the “Quick pick” feature of the mobile app, a new annotation is created (with its own intake) and linked to the original through parent_annotation_id. Each annotation therefore corresponds to a single meal event.

Intake

An intake captures a single consumption event of a dish: the date and time the user ate the meal (consumed_at) and the timezone of the device at that moment.

In practice, each annotation has exactly one intake. When a user re-logs the same meal via the “Quick pick” feature, the mobile app does not add a new intake to the original annotation — it creates a new child annotation (linked via parent_annotation_id) with its own intake and a fresh copy of the annotation items.

The consumed_at of an intake must fall within the participation’s started_at/ended_at window. Intakes feed the daily nutrient and HEI computations: whenever an intake is created, updated, or destroyed, the affected day(s) are scheduled for refresh.

Segmentation

When the dish is created with a picture, it is also associated with a segmentation object. The segmentation sends the picture to an automated segmentation service that creates the annotation items for each food detected.

Annotation

The annotation is a key object in the system as it links the dish, its intakes, a cohort, and the foods or products present in the dish.

It is associated with multiple annotation item objects that capture which foods or barcoded products are present in the dish. Annotation items are created by an automated segmentation service or by a human annotator (collaborator).

It can also be associated with multiple comment objects. The comments are used to communicate with the user and ask for information about the dish.

Lifecycle

A typical annotation goes through the following lifecycle:

stateDiagram-v2
[*] --> initial
initial --> awaiting_segmentation_service : picture sent to segmentation service
initial --> annotatable : no picture to segment
awaiting_segmentation_service --> annotatable : segmentation service performed task
annotatable --> awaiting_llm_service : user posts comment
awaiting_llm_service --> annotatable : chatbot answers
annotatable --> info_asked : annotator asks question
info_asked --> requires_human_intervention : user answers annotator's question
annotatable --> annotated : annotator finishes annotation
info_asked --> annotated : annotator finishes annotation
requires_human_intervention --> annotated : annotator finishes annotation
annotated --> annotatable : annotator clicks "Allow annotation"
annotated --> [*]

The state machine allows a few more transitions, left out of the diagram to keep it readable:

  • Escalation to requires_human_intervention is possible from any other state — including awaiting_segmentation_service when the segmentation service fails, awaiting_llm_service when the chatbot cannot answer, and annotated when a later analysis fails.
  • Reopening to annotatable is also possible from info_asked, not only from the states shown.
  • info_asked can also be entered from requires_human_intervention, when the annotator asks a further question.

  • awaiting_llm_service can also be entered directly from initial, when a user comments before the dish has been analyzed.

  • initial: The state of a newly created annotation, before the segmentation service has been called.
  • awaiting_segmentation_service: The state of an annotation after the segmentation service has been called. The segmentation service is an external service that takes a dish picture and returns a list of annotation_items objects that capture the location of the food in the dish picture. The segmentation service is called asynchronously, so the annotation is in this state until the segmentation service has finished.
  • annotatable: The state of an annotation after the segmentation service has finished. The annotation is now ready to be annotated by a human annotator, or commented on by a user (and answered by the chatbot).
  • info_asked: The state of an annotation after the human annotator has asked the user for more information about the dish. The user can then answer the question by adding a comment to the annotation.
  • awaiting_llm_service: The state of an annotation after a user posted a comment and is waiting for the chatbot to answer.
  • requires_human_intervention: The state of an annotation after a user answered an annotator’s comment, or when the user sends too many comments and the chatbot fails to make the appropriate changes to the dish, or when the segmentation service fails to analyze the image.
  • annotated: The state of an annotation after the human annotator has finished annotating the dish.

Once the annotation is in the annotated state, updates and commenting are disabled. To re-enable them, the annotation must be reset to the annotatable state by clicking on the “Allow annotation” button on the annotation page.

Annotation item

An annotation item links a dish to a food or product. It is associated with a PolygonSet object that captures the location of the food or product in the dish picture. It is also associated with an Item object that captures the food or product information.

Comment

A comment is a message attached to an annotation, used as the conversation thread between the user, the human annotator, and the chatbot. Every comment has exactly one author, which is either:

  • a user (typed in the mobile app)
  • a collaborator (typed in the annotation interface)
  • the chatbot (bot flag set), or a system message (welcome and kcal_verification are dedicated flags for the welcome message and the calorie-verification request)

A comment can be marked silent, in which case the recipient does not receive a push notification — typically used by annotators for acknowledgements that don’t require a response from the user.

Posting a comment also influences the annotation lifecycle: a user comment on an annotatable annotation hands it off to the chatbot (awaiting_llm_service), and a user’s answer to an annotator’s question moves it to requires_human_intervention.

Food list

A food list groups foods from the same source and version. It can be editable or not. Typically, a food list imported from an official source is not editable, while a custom food list created manually for a specific cohort is editable.

Food

A food is associated with a specific food list and might be editable depending on the food list it belongs to.

Product

MyFoodRepo caches product data in its own database. Product data comes from OpenFoodFacts. When a user scans a product barcode and the product does not already exist in the MyFoodRepo database, the user will be asked to take at least five pictures of the product, which are recorded together with the barcode in the database. The MyFoodRepo backend then looks the barcode up on OpenFoodFacts. If data exists for it, the data is imported into the MyFoodRepo database and the product is marked complete. If it does not, the product pictures are analyzed by the AI vision service, which extracts the nutritional data and attempts to match the product to a food; a product that cannot be resolved this way is marked incomplete.

A weekly job refreshes the list of barcodes known to OpenFoodFacts, so a product that was unavailable when it was first scanned can be recognized later. The nutritional data of an existing product is not re-fetched on a schedule; refreshing it is a manual operation run by the MyFoodRepo team.

Products imported before the OpenFoodFacts-only pipeline may still carry FoodRepo as their source. Product data was historically imported from FoodRepo, and product pictures were uploaded there for parsing, but that integration has been retired.