On the surface, it sounds like a straightforward AI project:

upload image → run model → return results

But once I actually started building it, I realized it’s not a single model problem.

It’s a pipeline problem.

And every stage has uncertainty.

System Overview

The system I built is split into 5 main stages:

Image input + preprocessing Object detection (food identification) Portion estimation Nutrition mapping layer Aggregation + output formatting

There’s no single “magic model” doing everything. Each stage handles a different part of the problem.

Image Input Layer

The frontend simply accepts an image upload.

But I added a few constraints early:

normalize image size before sending to backend compress large images (to reduce inference latency) reject extremely low-quality inputs (blurry / dark images)

This might sound basic, but it actually matters a lot because downstream model performance depends heavily on input consistency.

Food Detection Layer

This is where things start getting interesting.

Instead of treating the whole image as one object, I run a detection step to identify multiple food items.

The challenge here is:

single-item food is easy (burger, pizza) mixed dishes are not (biryani, thali, pasta bowls)

The model doesn’t always separate components cleanly, so I had to design the system assuming partial failure.

Meaning:

detection is never fully correct, but it should be “good enough” to continue the pipeline.

Portion Estimation (Hardest Part)

This was the most unreliable part of the entire system.

Images don’t give:

weight volume density ingredient ratios

So I had to rely on visual heuristics:

plate size estimation relative object scaling bounding box area approximation standard serving assumptions

At this stage, I stopped thinking in terms of “accuracy” and started thinking in terms of:

bounded error instead of exact values

Because exact values are impossible from a single image.

Nutrition Mapping Layer

Once I had:

detected food label estimated portion size

I mapped everything to a basic nutrition dataset.

This part is more deterministic compared to earlier stages.

Typical structure:

calories per 100g protein / carb / fat ratios standardized serving conversions

The output here is basically structured lookup + scaling.

Nothing fancy — just reliable data handling.

Aggregation Layer

After processing all detected items, I aggregate results into a single response:

total calories macro breakdown optional confidence adjustment simple structured output for UI

I intentionally avoided overcomplicating this layer.

The goal is not to present raw model outputs, but to return something usable.

Key Engineering Decision: Pipeline Over Single Model

The biggest design choice was avoiding a “single model solution.”

Instead of trying to train or fine-tune one model to do everything, I broke it into modular steps.

Why?

Because debugging becomes impossible otherwise.

With a pipeline:

I can isolate failure points I can improve stages independently I can swap components without breaking everything

This made iteration much faster.

Biggest Problem: Uncertainty Propagation

One issue I didn’t fully anticipate:

Errors compound across stages.

Example:

small detection error → wrong food label → wrong nutrition mapping → incorrect final output

So even if each stage is “80% accurate,” final output quality can drop faster than expected.

This forced me to rethink confidence handling later in the pipeline.

Performance Considerations

A few practical things I had to optimize:

image compression before inference batching requests when possible caching nutrition lookups limiting model calls per request

Latency matters more than people expect in consumer AI apps.

If it feels slow, users just stop using it.

What I Would Do Differently

If I rebuild this again, I would:

introduce confidence scoring per stage allow fallback “unknown food” states instead of forcing guesses add better dataset alignment for regional foods simplify portion estimation assumptions even further

Most mistakes came from trying to be too “smart” too early.

Final Thoughts

This project wasn’t really about food or nutrition.

It turned into a systems design problem:

handling noisy input building multi-stage pipelines managing uncertainty optimizing for speed over perfection

And the biggest lesson was simple:

real-world AI systems are mostly about handling imperfect data, not achieving perfect predictions.

Try it out

If you want to test the system in practice, you can check out SnapMacros and see how the pipeline behaves on real images.