Understanding Indonesia means reading the context behind a question: everyday terms, relevant institutions, and the needs of the person asking. For IsonAI, this is part of the development work we measure and refine continuously.
In an internal evaluation in August 2026, we compared responses to 1,000 questions across 56 categories before and after refinements to instructions and answer processing. The share classified as OK or WEAK increased from 67.9% to 89.6%, a gain of 21.7 percentage points.
This result shows progress on the behavioral criteria checked by our automatic grader. It measures performance against those diagnostic rules, rather than factual accuracy. This note explains what the result means and how it informs IsonAI’s development.
Indonesian context as a starting point
When users ask about public services, employment, education, or everyday matters, answers need to follow the context they intend. For relevant questions, IsonAI is directed to use Indonesian systems and terminology as a starting point.
The approach remains flexible. Users can specify another country or jurisdiction, and the assistant should follow that context. For universal topics such as mathematics or programming, the explanation stays focused on the problem itself.
This evaluation examines three aspects: local context when needed, substantive answer content, and uncertainty attached to the appropriate details.
A thousand questions through the conversation service
The question set was assembled for internal diagnosis and contains 56 category labels, including public services, employment, banking, education, culture, transport, and technology. Each question has a fixed ID, allowing before-and-after responses to be paired on the same task.
The runner requested IsonAI Omni through the conversation service. This examines the experience shaped jointly by application instructions, retrieval when used, and answer processing. The results describe the service in August 2026, separately from the PRME update and September’s model naming changes.
The publication audit confirmed 1,000 unique questions with matching text and categories in both groups. Retry records were consolidated by ID, keeping the last response under the comparison tool’s existing rule. Every percentage uses 1,000 questions per group as its denominator.
Four refinements
More targeted uncertainty. We adjusted instructions so references to knowledge limitations are used where appropriate. Uncertain details should receive the relevant explanation, while other parts can be answered directly.
Clearer local-context guidance. We clarified the use of Indonesian institutions, systems, and terms for relevant questions while respecting other contexts supplied by users.
Cleaner answer presentation. Output processing was refined so changes to content would not leave empty list or formatting markers.
Content protection during text cleanup. We added a check to prevent phrase cleanup from emptying a line that previously contained content.
These four changes were implemented as general rules, without adding answer keys for individual questions. They focus on system behavior that can apply across many tasks.
Evaluation results
The grader applies the same rules to both response sets. OK means that no flagging rule applied. WEAK means that the answer passed the preceding checks but still contained a detected stock uncertainty phrase.
Together, these categories rose from 679 to 896 answers. Separately, OK rose from 644 to 828 answers. OK and WEAK are evaluation-tool labels; neither establishes that every factual claim in an answer was verified.
| Automatic grader category | Before | After |
|---|---|---|
| OK — passed the flagging rules | 644 | 828 |
| WEAK — stock uncertainty phrase detected | 35 | 68 |
| GENERIC — no local marker detected in selected categories | 88 | 96 |
| ASKBACK — follow-up-question pattern in a short answer | 0 | 0 |
| EMPTY — length or remaining-content threshold applied | 0 | 1 |
| FOREIGN — foreign markers with few local markers | 0 | 1 |
| MEMORY — particular phrases about prior knowledge | 225 | 0 |
| ERROR — request recorded as an error | 8 | 6 |
| Total unique questions | 1,000 | 1,000 |
Under the comparison tool’s ranking, 283 questions moved to a higher category, 641 stayed in the same category, and 76 moved lower. This distribution helps prioritize further refinements, including WEAK and GENERIC, whose counts increased.
The largest transition was 201 questions moving from MEMORY to OK. This aligns with the instruction change concerning prior-knowledge phrases. Because MEMORY is a phrase detector, that change demonstrates the targeted language-behavior adjustment; gains in factual correctness require a separate assessment.
Reading the results within their scope
Diagnostic evaluation gives us a consistent way to identify patterns and inspect changes. Several boundaries accompany the results so their interpretation stays precise.
The questions were assembled for internal evaluation and used during diagnosis. They do not yet represent all users or provide an independent test on unseen questions. Vocabulary and length rules are indicators: valid local references can be missed, and short answers can hit a threshold despite containing information.
The comparison uses one pair of recorded result sets with retries. The four changes were introduced together, so their individual contributions have not been isolated. Generation variability and service conditions can also affect results. We do not yet report uncertainty intervals or comprehensive fact verification.
The next stage of measurement needs to complement these indicators with human assessment, a separate test set, and repeated runs. Factual correctness, contextual relevance, answer completeness, and service reliability need to be assessed as distinct dimensions.
Counting note
As a counting note, the baseline file contained 1,002 rows because two IDs included retry records. Both attempts for each duplicated ID were labeled ERROR. Consolidating by ID produced 1,000 comparison units and kept percentage denominators consistent.
For IsonAI, this progress reinforces an evaluation-led approach: observe answers, trace their causes, refine general rules, and check the results again. Understanding Indonesian context is ongoing work that we build through measurable steps.