# I'm afraid I don't know, Jacek. Jev, HAL 9000 and the return of machine learning > Jev in the cloud and Laya running locally on a Mac, tested on 30 tickets in Polish. Accuracy, traps, confidence and a model cascade - a practitioner's take. - Source: https://snok.ai/en/news/blog/afraid-i-dont-know-jev-hal-9000-machine-learning/ - Author: Jacek Bugajski - Published: 2026-09-25 --- In "2001: A Space Odyssey" HAL 9000 assures a BBC journalist that no 9000 computer has ever made a mistake. A few scenes later it tells Dave Bowman that the AE-35 unit is going to go 100 percent failure within 72 hours, and calls that a completely reliable figure. Bowman tests the unit and finds nothing wrong, and the twin 9000 computer at mission control concludes that HAL is in error. Stanley Kubrick's film was released in 1968 and, in a single thread, described a problem that has a name today: a model without calibrated confidence. I remembered that scene on 25 September, when I asked Jev what angle this text should take. Jev is a decision model from TypeSafe AI. It does not write answers; it returns a decision together with a probability distribution. I gave it three options. The top two scored 0.53 and 0.46, and the confidence of the whole answer was 0.30. In plain words: I'm afraid I don't know, Jacek. Dave never got that answer from HAL. ![Jacek's face reflected in the red lens of a retro-futuristic computer](https://snok.ai/images/blog/obawiam-sie-ze-nie-wiem-jacku-obiektyw.webp) *The illustrations in this post were generated by AI from a photo of the author.* ## The week machine learning came back TypeSafe AI went public on 15 September. Within days a whole ecosystem grew around Jev, and that interests me more than the model itself. [Laya](https://github.com/NandhaKishorM/laya), an open model with the same contract (a choice from a list, a score on a rubric, the probability that a statement is true), runs on ModernBERT and mmBERT encoders, the descendants of 2018's BERT. There is already a [port to MLX that runs on a Mac with no cloud](https://aiidelist.com/blog/what-is-laya-mlx). Together AI published a [guide to fine-tuning your own Jev-style classifier](https://www.together.ai/blog/how-to-train-your-own-jev) on Qwen3.5 4B, and the Laya authors shared a notebook for fine-tuning on free Kaggle GPUs. Anyone who did data science before ChatGPT will recognise the kit at once: an encoder, a classification head, calibration, a decision threshold, a test set. Since ChatGPT arrived we have handed more and more decisions to models that write. Within a week the market remembered that a decision and a text are two different tasks, and that we have long had better tools for the first one. ## How I use Jev day to day Since 20 September Jev has been my second opinion. Whenever my AI assistant makes a judgement, such as whether a post is clear or which option to pick, Jev's assessment with its full distribution sits next to it, and every disagreement goes into a register. I only send public or synthetic content to the API in the US. Client data never leaves our infrastructure. For the post about [SAPMAP](https://snok.ai/en/news/blog/sapmap-bloodhound-for-sap/), Jev rated readability for a non-technical reader almost as a tie: 0.51 for "clear to a decision-maker" and 0.47 for "partly clear". We added an explanation of what RFC is. When rating the urgency of one topic it returned a score of 1.35 at a confidence of 0.03, because it could not decide between "this quarter" and "this month". The label alone would have said "medium urgency", and nobody would have noticed that the model was guessing. ## A test in our lab A few single calls are an anecdote, so on 25 September we ran a test. Thirty synthetic tickets in Polish and one question: which team should take the case - SAP, cybersecurity, automation, or nobody because it is out of scope. Among them we placed three traps we know from tender search, where SAP stands for System Alarmowy Pożaru, the Polish acronym for a fire alarm system, and four deliberately ambiguous cases, for example a UiPath robot that stopped working after a SAP GUI update. The same questions went to Jev in the cloud and to two versions of Laya running locally on a Mac with an M4 Max chip. I wrote about why local hardware matters to us in the post on [Mac Studio as a POC machine](https://snok.ai/en/news/blog/mac-studio-512gb-vs-dgx-spark-poc-machine/). ![Terminal capture: three tickets through Laya locally and Jev over the API, with the choice, confidence and response time](https://snok.ai/images/blog/obawiam-sie-ze-nie-wiem-jacku-terminal.webp) **Jev** got all 26 unambiguous tickets right and caught all three traps, with a median response time of about 0.66 seconds. Average confidence was 1.00 on clear tickets and 0.73 on ambiguous ones, so it drops where it should. All 50 calls, 24,387 input tokens in total, cost about 0.001 USD at the vendor's price list. **Multilingual Laya without fine-tuning** got 12 of 26 tickets right and caught none of the traps: it sent all three fire alarms to the SAP or cybersecurity team. In exchange it answers in 8 milliseconds without sending anything off the machine. More interesting is how it fails. Its average confidence is 0.74 on correct answers and 0.19 on wrong ones. The model guesses poorly, but it knows well when it is guessing. One caveat: the Laya authors state plainly that the multilingual version has no fitted calibration, so these numbers describe the model before tuning. **English Laya on Polish text** got 13 of 26 right at a confidence of about 0.11 on every ticket, so it did not pretend to understand the language. ## A threshold that works one time and not the next One ticket taught me the most: "An AI agent should read incident tickets and assign them a priority". I sent it to Jev 21 times. The answer was the same every time, automation, but the confidence ranged from 0.45 to 0.63. With a 0.60 threshold in the process, the same text would have passed automatically three times and gone to a human eighteen times. Local Laya returns the identical 0.41 on that ticket every time. A UiPath practitioner who [compared Jev with an agent in Maestro Flow](https://www.linkedin.com/feed/update/urn:li:activity:7508514827871375360/) on ten cases described a similar lesson on LinkedIn: the model returned a confidence of 1.00 in eight of them, so the "below 0.60 goes to a human" gate almost never fired. We do not set the hand-off threshold by feel. We set it on a validation set, looking at the confidence distribution and at the cost of each error. The Laya documentation puts it this way: "A threshold is a policy you choose from measured accuracy at that coverage on your data, not a property of the model." That principle has been in every machine learning textbook for twenty years. ## A cascade, an old idea in a new garden These numbers point to an architecture data science teams used long before language models: a cascade. A cheap local model decides where it is confident and passes the rest on. In our test Laya with a 0.5 threshold settled 12 of 30 tickets locally and got one wrong. The other 18 went to Jev. The result was 29 correct decisions out of 30, and only 18 tickets left the machine instead of 30. With client data this split matters more than milliseconds, because we do not send NDA material to a US API at all. The Laya authors report that fine-tuning on decisions from a specific domain raises their model's accuracy on a typed-decisions benchmark from 0.362 to 0.766. That is an old truth too: your own well-labelled data beats a general model. But you have to build the labelled set yourself, and no new model will do that work for you. ## Jev in my garden of tools In my setup Jev joins a garden where every tool has its own bed. Regular expressions stay as a permanent fallback: when the API does not answer, the system falls back to rules instead of stopping. Local models handle data that cannot leave the client's infrastructure. That is the same direction in which we build [on-premise LLMs at SNOK](https://snok.ai/en/offer/ai-automation/llm-on-premise/), including on hardware such as the [Lenovo ThinkStation PGX](https://snok.ai/en/news/blog/lenovo-thinkstation-pgx-gb10-hands-on/). The large language model stays with writing and multi-step reasoning. Jev gets the small decisions in between: classification, routing, quality gates. The rule is simple: the model proposes a branch, and code decides whether it may be taken. Publishing, payments and deleting data always go through a hard check. On the lab machine where we experiment with the Hermes agent, there is a plan for tender search. Jev will first run in shadow mode, deciding and only logging its decisions, and we split the corpus into a tuning part and a frozen test part, with a required recall of 0.95. Honestly counted, confirming such a result will take three to four months. For now it is a plan, not a deployment. HAL could not say "I don't know". Jev and Laya can, and a confidence of 0.30 is a legitimate answer for them, one that code can act on. Where in your systems does a language model make a yes-or-no decision today? I am curious whether you also have more such places than you assumed. ## Sources - TypeSafe AI, product page and Jev API documentation - https://typesafe.ai/, https://docs.typesafe.ai/api.md (accessed 25.09.2026) - Laya, repository and README with the typed-decisions benchmark and the calibration note - https://github.com/NandhaKishorM/laya (accessed 25.09.2026) - laya-mlx, Laya port to Apple MLX - https://github.com/mizorewww/laya-mlx (accessed 25.09.2026) - AI IDE List, "What Is Laya-MLX?", 20.09.2026 - https://aiidelist.com/blog/what-is-laya-mlx (accessed 25.09.2026) - Together AI, tev1 and the "How to train your own Jev" guide - https://github.com/togethercomputer/tev1, https://www.together.ai/blog/how-to-train-your-own-jev (accessed 25.09.2026) - Jev and UiPath agent measurement in Maestro Flow, LinkedIn post - https://www.linkedin.com/feed/update/urn:li:activity:7508514827871375360/ (accessed 25.09.2026) - SNOK in-house test, 25.09.2026: 30 synthetic tickets, Jev jev-1.13.0 over the API, laya-mlx 0.2.0 on a Mac with M4 Max - "2001: A Space Odyssey", directed by Stanley Kubrick, 1968