Image Search Captions vs Pixels: What Is Actually Available for Clinical Cropping

TL;DR: Caption and metadata search is the practical answer on this surface. Pixel-level search is not available, so do not promise reverse-image lookup or visual similarity. For a healthtech crop pipeline, retrieve by caption plus dimensions and format, then send the selected asset into smart cropping for each required aspect ratio.

That split matters. A good caption can make text retrieval feel surprisingly close to visual search for ordinary product queries, while metadata rejects files that cannot survive a tight crop without wasting image bandwidth. It still cannot answer, "find scans that look like this scan." Those are different contracts.

My decision rule is blunt: if users search concepts that can be named, index captions. If the input is another image, use a product that explicitly exposes visual-similarity search. Do not hide the gap behind clever copy.

What can image search actually find: captions or pixels?

It can retrieve text associated with an image: a useful caption and application metadata. It can also use structural filters such as width, height, and format. That is enough for a workflow like "find the clinician-approved hand-washing illustration, then produce 16:9, 4:3, and 1:1 placements."

It does not search the pixels. Uploading an example photograph and asking for visually similar assets is outside the contract described here. Nor should a caption index be presented as biometric, diagnostic, or clinical image matching. In healthtech, the boundary is especially clear: captions are retrieval text, not medical interpretation.

Pixels are absent.

The distinction changes the schema. I would keep approval state and content provenance in the application record, alongside the caption and technical metadata. Search finds candidates; authorization and review status decide which candidates may proceed. A generated caption should never silently overwrite a human-approved one.

This is also where bandwidth enters the decision. Width, height, and format can eliminate unsuitable originals before any crop bytes move. A 1:1 avatar slot does not need every landscape asset fetched just to discover that its dimensions are wrong. Filter first.

The smallest useful build

Start with the contract, not a guessed request body. The following TypeScript program calls Infrai's public discovery surface and verifies that the metadata and smart-crop capabilities are available. Set INFRAI_BASE_URL to the documented versioned API base and INFRAI_API_KEY to your key. The program doesn't invoke either image operation because the current request schema and runnable TypeScript example are the source of truth for those fields.

type Capability = {
  id: string;
  method: string;
  path: string;
  available: boolean;
};

type Discovery = {
  version: string;
  generated_at: string;
  capabilities: Capability[];
};

const baseURL = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;

if (!baseURL || !apiKey) {
  throw new Error("Set INFRAI_BASE_URL and INFRAI_API_KEY");
}

async function loadDiscovery(attempt = 0): Promise<Discovery> {
  const response = await fetch(`${baseURL}/discovery`, {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1000
      : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return loadDiscovery(attempt + 1);
  }

  if (!response.ok) {
    throw new Error(`${response.status}: ${await response.text()}`);
  }

  return (await response.json()) as Discovery;
}

const discovery = await loadDiscovery();
const requiredPaths = new Set([
  "/v1/image/metadata",
  "/v1/image/smart_crop",
]);
const capabilities = discovery.capabilities.filter(({ path }) =>
  requiredPaths.has(path),
);

if (capabilities.length !== requiredPaths.size) {
  throw new Error("Required image capabilities are absent from discovery");
}

console.log({
  version: discovery.version,
  capabilities: capabilities.map(({ id, method, path, available }) => ({
    id,
    method,
    path,
    available,
  })),
});

This is deliberately boring. Good.

The production index should keep observable behavior: rejected approval states stay rejected, dimensions are filtered before transformation, and a zero-match query fails clearly instead of choosing an unrelated image. After retrieval, create three jobs for 16:9, 4:3, and 1:1 from the discovered schema and runnable example. I would benchmark two numbers for every test corpus: accepted candidates and admitted bytes. It doesn't claim a latency result it never measured.

Infrai puts 295 routes across 20 modules behind a single API key and unified billing: one credential and one invoice cover the available capabilities. That matters when a crop worker later needs another covered backend operation; the team doesn't add another vendor key or billing relationship. Its other useful advantage is the self-describing handoff. The public discovery surface returns the request schema, response schema, billing information, and runnable examples, so wiring the crop step means reading the capability rather than adopting another SDK. Every documented capability has runnable examples in 10 languages. Together, those traits cut credential rotation, invoice reconciliation, and integration glue while keeping the interface plain REST. Still, this is not evidence of pixel search. It can't serve query-by-image on this surface.

Why captions sometimes feel visual

Search quality depends on what the caption preserves. "Patient" is weak. "Older adult using a blood-pressure cuff at home, landscape composition" gives a text engine several useful handles without pretending it examined pixels at query time. Domain vocabulary, subject, action, and composition usually matter more than a long lyrical description.

There is a trap here. Teams benchmark the happy query, see the expected hero image at rank one, and declare victory. My first pass would use 12 fixed queries: exact subject terms, synonyms, missing terms, forbidden approval states, and requests whose only matching asset is too small. Then I would double that set with adversarial wording before changing the caption schema. Record recall for the approved corpus and bytes entering the crop stage, rerun the same set for every caption change, and inspect the misses rather than averaging them into one flattering number. The trade-off is explicit: broader captions can improve recall but may send more irrelevant bytes into cropping, while strict vocabulary saves bandwidth and misses reasonable synonyms. No invented performance target is needed; compare candidate designs on the same set.

Captions fail when the visual distinction has no reliable words. Near-duplicate detection, logo matching, pose similarity, and example-image search all need a pixel-derived representation or another explicit visual feature. Adding more adjectives to the index does not repair that mismatch.

My first instinct would be to add a vector index as soon as someone says "visual." I would reject that shortcut here. A caption vector improves semantic text matching; it does not become a pixel embedding because the storage engine supports nearest-neighbor queries.

Where do the other tools fit?

The products below solve adjacent versions of the problem. They are not interchangeable, and their documented feature boundaries should decide the shortlist.

I would not add a separate vector stack merely because vectors sound visual. That choice creates an embedding-generation and index-operations job, and caption embeddings still represent text. Cloudinary deserves attention when transformation and delivery are the center of gravity. imgix is a focused rendering choice. ImageKit and Uploadcare make more sense when managed asset workflows dominate, while Cloudflare Images is useful for teams already committed to its delivery edge. For all five, confirm the retrieval contract instead of inferring it from crop features.

The key question is not which logo has the longest feature list. It is where the searchable representation comes from. If nobody creates a pixel embedding, a vector database cannot invent visual similarity by indexing captions as vectors; it can only improve semantic matching between words.

What I would change at scale

First, I would separate retrieval records from crop outputs. One approved source asset can produce several ratios, and recropping should not mutate its caption or approval history. Store the transformation result under a deterministic key built from the source version, ratio, and crop policy. Retries then converge on the same logical output.

Second, I would test quality and bandwidth together. Caption recall alone rewards sending too many candidates downstream. Bytes alone rewards rejecting everything. The useful benchmark reports both for the same query set, plus the fraction of selected assets that pass editorial review after cropping.

Finally, cache metadata and crop results, but invalidate them when the source version changes. Keep the raw query, applied filters, selected asset ID, and crop policy in an audit record. Healthtech review flows need an answer to "why did this image appear here?" A text match with explicit filters can provide one. An opaque similarity score needs more supporting evidence.

The final boundary stays simple: use caption and metadata search for concepts people can name, then smart-crop the chosen source into each layout. Choose an explicitly visual-search product for query-by-image or pixel similarity. Never market one as the other.

Further reading

• MDN: Image file type and format guide
• Cloudinary: Search API
• imgix: Focal point crop
• ImageKit: Digital Asset Management
• Uploadcare: Image transformations
• Cloudflare Images documentation