The photospike

What's a photospike?

A spike is a quick, throwaway build. This one was aimed at analysing my photos. The plan was to feed them to a local vision model (like the LLMs behind AI chatbots, but it looks at pictures instead of words) and get back a written description of what's in the photo, how it's lit, the mood, what a viewer might miss. The idea was to someday run my entire library of 10,000+ photos through it to try and discover patterns.

The first run

The first trial was 23 iPhone shots from a holiday, taken with Mood - a film-emulation app that bakes grain and colour grading in as you shoot - all in the ultra-wide XPan ratio (65:24). I ran them through gemma4:31b locally, via Ollama, at a low temperature (0.1) so repeated runs wouldn't wander and I could compare prompts fairly. A model that size is slow on a laptop - nearly two minutes per photo - but for judging output quality that didn't matter yet.

The results were mixed. On the obvious things it was sharp - it named the Sydney Opera House without hesitating, and caught details I'd have missed myself, like the faint silhouette of a distant island. But reading all 23 outputs field by field, the misses repeated in patterns:

• the season was hemisphere-blind - Australian autumn kept coming back as "spring"
• the location field invented countries whenever the photo gave it nothing to work with, e.g. an overcast Queensland beach was identified as Ireland
• composition_type came back "panoramic_sweep" for 71% of the photos - I'd told the model these were XPan panoramas, and it echoed that straight back at me
• depth_impression was "deep" for ~89% - the definition wasn't specific enough so almost everything got labelled as "deep"
• "Kodachrome" turned up in 17 of 23 style notes, and the titles kept reaching for the same handful of words

It wasn't surprising - the model doesn't know any better. The locations were guesses I'd forced by holding back the GPS. The panoramic-sweep obsession was just my own wording coming back at me. The model was fine - the prompt and the schema were what needed fixing.

Inject, don't assert

The second version worked the other way round. The camera already writes plenty into every file's EXIF: colour temperature, the lens, exposure, the date and GPS coordinates, even the film stock Mood was emulating. This information could be extracted and injected straight into the prompt.

So the next output split into three tiers:

• Read from the file. Colour temperature, lens, exposure, GPS-derived place, capture time, hemisphere-correct season - handed to the model as facts.
• Pick from a list. scene_type, composition_type, depth_impression, energy_level - categorical fields with a fixed vocabulary, so a whole library of photos charts cleanly.
• Say something. mood, the_overlooked_detail, beyond_the_frame, least_certain - free text, the fields where I actually want the model's unique read of the photo.

The title prompt carries a banned-word list - quiet, silence, breath, echo, weight - the words the model reached for whenever it wanted to sound deep.

Some of the fields are really there to test the model itself - can it read a sign, guess the time of day, say what's just out of frame? My favourite is a field called the_overlooked_detail - it asks to describe the single small thing a viewer might ignore. It's the one field whose answers read like the model actually looked, instead of just filling a box.

Eight runs, one prompt

Alongside tuning the prompt, I wanted to see how much the model choice itself mattered - and whether something smaller could replace the two-minutes-a-photo 31b. So I set up a benchmark. I ran the identical prompt through a grid: two models (gemma4:e4b and gemma4:26b-A4B) × two quantisations (Q4 and Q8 - a quantisation is a compressed model, trading precision for size and speed) × two temperatures (0.2 and 0.8). Each photo came back with 27 generated fields. Each run now covered 232 photos rather than the first 23, and all eight together took five to seven hours on an M4 Max.

The scoreboard

Only the model and the temperature changed across the eight runs, so I could tell whether each failure came from the model or from me.

• Depth nuance tracks model size. The small e4b calls 86-94% of photos "deep"; the bigger 26b holds the line at 76-85%.
• The composition bias shows up in all eight runs - so the fault is mine: the categories I wrote are too loose.
• The "Autumn..." title tic is a small-model habit. e4b led 28-81 titles a run with the season; the 26b, only 1-4.
• Temperature barely helped. From 0.2 to 0.8 it didn't fix the formulaic titles and made the small model sloppier - e4b-Q4's invalid values went from 4 to 11. Which model you pick matters far more.
• Q8 beat Q4, most on the small model - a 4B has less spare capacity - compressing it throws away things it was actually using. The cleanest was 26b at Q8: 1-2 bad values out of 232, against 10-11 for the Q4s.
• Time of day is hard for every model, and size doesn't help (28-43% agreement with the real capture time). The answer mostly isn't in the pixels - which is why the field stays in as a test, graded against the EXIF timestamp.

Overall winner: gemma4:26b-A4B at Q8 - cleanest output, best depth nuance, none of the title tic, and a fraction of the 31b's runtime.

What's next

Although the initial idea was to run it against all 10,000+ photos in my library, I'm not doing that yet. The prompt matters more than the model choice, and it needs more fine-tuning: making "layered" composition mean one specific thing, stopping every title from opening with the season, and sorting out time_of_day.

All eight runs are already loaded into a dashboard on this site, where you can flip through the photos and compare the models yourself. Building it became a project of its own, which is where the next article picks up.