Turkish version: Medium
I run likethisgame.com. Every game has a page, and that page lists other games someone who liked it might also like. A language model writes those recommendations and I store them.
When I looked at the database, I found a problem. Black Myth: Wukong was on one in every ten pages that had recommendations. It had been recommended for a music game, for a sports game, for a puzzle game.
When I dug into why, it turned out my own code was steering the model toward those games. Each time I asked for recommendations, I put twenty recent games in front of the model. When I could not find anything similar, the most popular games filled that list. So for games that had nothing in common, I kept showing the model the same candidates.
To pick that list better, I tried Jev, a decision model from TypeSafe. I did not have Jev write recommendations. I asked it to rate candidate games, then handed the model that writes the recommendations a list built from those ratings. I went step by step with small side-by-side tests. Some changes I kept, some I dropped.
Contents
- Why always the same games?
- What is Jev?
- How do I ask Jev?
- What if I filter out low-scoring recommendations?
- How does the new system work?
- What did I try, what did I keep?
- What did it cost?
- What is still unsolved?
1. Why always the same games?
The list I sent to the model had about six similar games and twenty new ones. The similar games came from the catalog I use. The new ones were picked by my code. So most of what the model saw was recent releases.
The code first looked at what the games had in common. When it found no match, it picked the ones with the most user ratings. So whether I asked for a music game or a puzzle game, the model got the same twenty popular games.
Black Myth: Wukong, for example, had been recommended on the pages of 12 music games and 10 puzzle games. Helldivers 2 and Balatro showed up on pages unrelated to their genres too. Of the 16,311 pages I checked, 1,724 recommended one of these popular games on a page from an unrelated genre.
2. What is Jev?
Jev is a decision model built by TypeSafe. You send it some text and questions about that text, and instead of writing a long answer it returns probabilities. In my case the question is: "Would someone who likes this game also like that other game?"
I think of it as an if that can read. Where I would normally write conditions by hand in code, I tell Jev what to look at. What to do with the result is still up to the code.
There are three kinds of question:
-
noul: the probability that a condition holds. This is what I used to rate how well games fit each other. -
choice: a probability for each option you give. You could ask, for example, which category a piece of content belongs to. -
score: a rating on a scale you define. For example, levels like low, medium and high.
One distinction matters here. A noul of 0.5 does not mean "these two games are moderately similar". It means that, from the text you gave, yes and no are about equally likely. And getting a number back does not by itself mean the answer is reliable; you have to try it on your own examples.
I reached Jev through OpenRouter's Decisions API. It is a separate endpoint from the usual chat requests. You pay per token you send; the answer itself is not billed. I checked the cost against the usage.cost field of my own calls.
3. How do I ask Jev?
I do not need a separate request for every candidate. I put the game I want recommendations for and twenty candidates into the same request, then add one fit question per candidate.
The request looks like this. I have shortened the question and criteria text:
const body = {
model: 'typesafe/jev-1.13',
state: {
source: sourceGame,
candidates: candidateGames,
},
questions: {
c0: {
type: 'noul',
instructions: '<does candidates[0] fit a fan of source>',
criteria: {
true: '<what a fit looks like>',
false: '<what a miss looks like>',
},
},
// One question per candidate: c1, c2, ...
},
};
const res = await fetch('https://openrouter.ai/api/alpha/decisions', {
method: 'POST',
headers: {
Authorization: `Bearer ${apiKey}`,
'Content-Type': 'application/json',
},
body: JSON.stringify(body),
});
const { answers, usage } = await res.json();
sourceGame and candidateGames hold each game's name, release year, genres and description. In the response, answers.c0.noul is the result for the first candidate and usage.cost is what the request cost.
I also checked whether asking about twenty games at once changes the result. I sent the same games one at a time as well. The two rankings largely agreed, with a Spearman correlation of 0.884. So I cut the number of requests to one twentieth. That does not mean the cost dropped to one twentieth; I did not measure that ratio.
A 600-candidate pool takes 30 requests. Sending eight requests at a time, rating one game's candidates took about two seconds and cost $0.007.
4. What if I filter out low-scoring recommendations?
My first idea was to have Jev check the recommendations the model had written and drop the ones with low scores. I tried it on the recommendations from two test runs on 30 games. But the unrelated games I wanted to remove did not always score low. When I set the cut strict enough to remove them, it also removed 150 of the 216 recommendations.
So I moved Jev to a different place. Instead of checking recommendations after they were written, I had it rank the candidates before the model saw them. The model would then write its recommendations drawing on that list.
5. How does the new system work?
Say I want games like Zen Bound 2. First the code finds 600 games in the catalog with a close genre or theme. These are not recommendations yet; they are the list to choose from.
Then I send those games to Jev and ask, for each one, "would someone who likes Zen Bound 2 like this too?" I rank the games by the answers and hand the top 30 to the other model. That model recommends four games, drawing on that list, and writes why it recommends them. So Jev rates the candidates, and the other model writes the recommendation you read on the page.
Before keeping the top 30, the code adjusts the scores a little. Games with a large number of user ratings, and games already recommended to other pages in the same weekly run, lose a small amount. This makes it harder for the same few games to keep rising to the top and leaves room for other candidates.
In the first trial, Zen Bound 2 got Braid, Lara Croft Go, Monument Valley II and Gorogoa. Preparing the page cost less than a cent in total.
6. What did I try, what did I keep?
I ran five small comparisons. The first four used the same 30 games. I read the two sets of recommendations side by side and picked the one I preferred, and only afterwards looked up which list had been made with which method. I had no separate scoring rubric. For the last test, 30 games were chosen from different genres and popularity levels. Some of them were older games I did not know well, so an AI agent did that comparison, again without knowing the methods.
First I replaced the old candidate list. The genre-matched list ranked by Jev did better on 11 games and worse on 9; on 10 I could not pick a clear preference. The popular games from unrelated genres that I had seen on 7 pages with the old method did not show up in this sample with the new one. But since I changed both the candidates and the ranking, I cannot count this as Jev's success alone.
I tried changing how the list is presented. Instead of mixing the chosen games into the existing list, I gave them as a separate ranked list in the prompt. Results improved on 8 games, got worse on 2, and stayed the same on the rest. I kept the separate list.
I gave 10 candidates instead of 30. Some results improved: a racing game's recommendations picked up Burnout 3 and Burnout Revenge. But on other pages the freed slots were filled by popular games like The Witcher 3 again. Overall it was a draw, 9 better and 9 worse; I left the list at 30.
I lowered the scores of popular games a little. This was the clearest gain: better on 9 games, worse on 2. Omori got EarthBound. On the Red Dead Redemption page, Sleeping Dogs and Days Gone took the place of The Witcher 3. Wukong dropped out of Ōkami's recommendations. I kept this adjustment too.
Finally I cut the 600 candidates down to 300. Jev's cost halved, but in the comparison the recommendations made with 600 candidates were preferred on 14 games and the ones made with 300 on 8; the remaining 8 were a draw. The smaller list lost some candidates that suited racing, sports and puzzle games in particular. The saving was about a third of a cent per page. I stayed with 600.
7. What did it cost?
The first weekly run prepared recommendations for 243 games. The total was about $2.02, of which $1.59 went to Jev and the rest to the models that write the recommendations and the short intro text at the top of each page. The cost per page stayed under a cent.
The interesting part was that most of the money went to rating the candidates, not to writing the recommendation. In the tests, Wukong dropped out of Ōkami's list and Red Dead Redemption got Sleeping Dogs. I got better results than with the old system, so I chose the new one. At under a cent per page, I found that a fair price for it.
8. What is still unsolved?
Not every game got better. With 600 candidates, for example, Tetris got Braid and The Witness. Both are puzzle games, but not what I expect when looking for something like Tetris. The 300-candidate run gave Puyo Puyo Tetris, Lumines and Meteos instead. I have not looked into why yet.
The repeats are not completely gone either. In the first run, Slay the Princess was recommended on 19 pages and Nine Sols on 16. While looking into that, I found another problem: the code that reduces repeats was counting every recommendation ever made. A game like Hollow Knight, recommended a lot months ago, was being pushed down in new lists as well. I changed the counter to reset at the start of each weekly run. I will check what that does to real pages in the next run.
There is also a comparison I have not made yet. When I changed the candidate list, I added Jev at the same time. What if I had given the same candidates without Jev, with a simpler ordering? Maybe a good part of the improvement comes from finding better candidates alone. To separate the two, I need to run that test as well.
Finally, the old recommendations did not fix themselves. The 1,724 pages I found are still live in their old form. Before regenerating all of them, I will try 30 pages and look at the results.
For now, Jev stays in the system. On the pages I tried, I got better recommendations than before. Whether I could get the same result with a simpler method is the subject of the next comparison.
Stay in touch: arasmehmet.com
Sources: TypeSafe documentation, OpenRouter Decisions API, Jev on OpenRouter. The experiment logs and evaluation files are in the project's private repository; they are not public.
