I built StacksNG for the Africa Deep Tech Challenge 2026: an offline coding assistant grounded in the docs of four Nigerian fintech APIs: Paystack, Flutterwave, Monnify, Termii. It runs on retrieval, not memory (RAG, if you know the term): pull the real doc passage before answering, cite the source. 780 scraped chunks, running entirely on-device via llama.cpp, the engine that runs the model locally. No cloud, no API keys, no hallucinated endpoints in payments code.
I didn't make the semifinals.
No feedback came with that. So I went looking myself.
StacksNG shipped qwen2.5-coder-7b, stock, compressed down to fit in memory (Q4_K_M quantization). That's the largest model that fits the challenge's 8GB envelope, and it's a deliberate bet against the scoring formula, not an oversight. I wrote the ablation into the report myself. A smaller, 1.5B version of the same pipeline wins the formula by about 35 points: faster, lighter, better on the speed and memory-efficiency scores. I rejected it anyway, because on one of my two registered test prompts, the 1.5B opened with:
import flutterwave
client = flutterwave.Client(...)
That library doesn't exist. The 7B, same retrieval context, wrote real requests calls against real documented endpoints. I picked correctness over the formula and said so in the report. Fine engineering decision. Turned out to be the wrong thing to be defending.
I cloned all 20 published semifinalists (full git history, in case anyone privatizes their repo after judging) and read every technical report. SME-Ledger, Jamii Afya, TaxSabi, CodeFellow, ARIS, Homa, all of them: LoRA, QLoRA, distillation, or a merge. Every single one touched the base model somehow.
StacksNG was the only stock-model architecture in the batch. I noticed the pattern, filed it under "interesting," and moved on. That was the first mistake.
My own submission.json:
"accuracy": []
Empty array. Not a low number. No number. git log on that file shows exactly one commit, never touched again before the deadline. The adtc-profiler run that produced it either skipped the accuracy gate or lm_eval wasn't installed at the time. Same failure mode either way: silent, not an error.
Sacc is 50% of the scoring formula. I'd shipped half the score as zero without meaning to.
Trying to fix that, I installed lm-eval-harness and pointed it at my own model through a local llama-server. It crashed. Every one of 200 requests came back "Invalid logprobs data."
Two bugs, both confirmed by reading the actual source:
-
adtc_profiler's ownaccuracy.pycallslm_evalwithbase_url=local. Not a URL. lm-eval's GGUF backend needshttp://host:portand POSTs to{base_url}/v1/completions. As shipped, that line fails for anyone who runs it literally. -
lm-eval-harness's GGUF backend expectsecho=trueto return logprobs for the whole echoed prompt, the old OpenAI completions behavior. Currentllama-serveronly returns logprobs for newly generated tokens. The harness hasn't caught up.
I patched both locally. Instead of trusting the broken echo behavior, I forced the model to generate the exact answer text (a grammar rule constraining what it's allowed to output) and read its confidence scores off that generation directly. Ran the hackathon's own default benchmark, a standard general-knowledge multiple-choice test. Got a real number:
arc_easy acc_norm = 0.74
Competitive. Ahead of one fine-tuned semifinalist, just behind two others. Plugged into my own formula: Stotal goes from the submitted 10.23 to 47.23. A 4.6x swing from one missing number, no architecture change.
I thought that was the end of it.
Your submission's originality score did not meet the threshold required to advance.
Originality Score (0–10): 3
Model Origin (Originality Review): Stock model, used as-is: Qwen2.5-Coder-7B-Instruct-GGUF from lmstudio-community's official quantization.
Round 1 runs an originality gate before any technical scoring even happens. Template compliance: fine. The RAG architecture, the citation grounding, the 780-chunk corpus: not mentioned. The problem was never Sacc. The problem was that I never touched the model.
All the archaeology (the broken base_url, the echo/logprobs mismatch, the 0.74) was real, reproducible, and completely beside the point. I'd spent a week debugging the wrong layer.
The pattern was sitting in my own data the whole time. Eighteen other teams fine-tuned. I read that fact, found it interesting, and filed it as a scoring-formula tradeoff instead of what it actually was: everyone else had cleared a bar I didn't know existed.
That's the uncomfortable part. The evidence was already in front of me before the rejection email arrived. I reasoned right past it anyway.
I was looking at the parts of the system I could inspect: the formula, the profiler source, the tooling. The part I couldn't see was the one that mattered: a review that runs above all of it, undocumented in any code or rule I had access to.
A RAG layer grounding a stock model is a defensible product decision. It is not, to an originality reviewer, evidence that you built anything. Those are different bars, and clearing one says nothing about the other.
If you're entering a hackathon with a scoring formula published anywhere: read it. Optimize against it. But ask what isn't in the formula too. A judging process can have a gate that never shows up in the rubric or the tooling. The only way to find it is to assume it exists and ship something that survives it anyway.
Ship a real modification. Even a light LoRA pass on your own corpus would have cleared this. The application layer was good work: retrieval, citations, the deterministic-gate bugs I fixed after a DEV.to reader found them in production. None of it was ever going to be evidence that the model itself was mine.
Good luck to the eighteen who advanced. You did the part I skipped.
StacksNG: github.com/dannwaneri/stacksng. Two tooling bugs found during this postmortem, filed upstream: adtc-profiler#4 and lm-evaluation-harness#4158.
