LLM metrics and advanced concepts

Metrics

Metrics across AccuLLM are averages, counts or sums across LLM engines, depending on what fits the metric.

For example, the prompt list shows the average rank for a prompt across engines. Click the arrow next to a prompt to view the ranks per engine.

You can filter by a single engine using the LLM engine filter. A switch in the top right corner on all pages also lets you select an engine. When this filter is active, all statistics in the platform reflect only the selected engine.

General metrics

These metrics appear in several places: the prompt list, the dashboard and the Competitors tab.

Rank

Your brand's rank is based on when it is mentioned in the prompt response. You are ranked against the competitors defined for your domain, including unpinned ones. A domain can have up to 50 competitors. By default, AccuLLM adds up to 20 competitors to a new domain, which you can manage during brand creation and later under the Competitors tab.

A brand is detected in one of two ways:

• The brand domain is present in the text. This match is case-insensitive: the spelling "adidas.com" matches "Adidas.com is a great website".
• One of the defined brand spellings appears in the text. These matches are case-sensitive: the spelling "foobar" does not match "FooBar is a great product".

To be included in ranking, brands must be defined as competitors. You can add, edit, delete or pin competitors in the Competitors tab. Adding or editing a competitor re-evaluates the latest day's responses; older responses are not re-evaluated.

A simplified example. Your brand is Nike, and your competitors are Adidas and Puma. The response reads: "Some of the best sneakers are made by Reebok, but Adidas also makes very nice sneakers. Nike's sneakers are also awesome. The sneakers from Asics are also great, and the same can be said for Puma." Reebok and Asics are not defined as competitors, so they do not count. Adidas ranks 1, Nike 2 and Puma 3. If Reebok is later added as a competitor, it only affects the latest day's and future responses.

Visibility

Visibility is 0% or 100% per engine, based on whether your brand appears in a prompt response. A 0% visibility means no rank. In the prompt list, the visibility shown is the average across engines: if your brand is visible in 3 out of 4 engines, visibility is 75%. Each engine is weighted equally. For statistics on a specific engine, apply the LLM engine filter.

Sentiment

After ranking, we analyze the sentiment of the text related to your brand. This includes both direct mentions and the surrounding context. If your brand appears in a list, the heading may be included; if it appears in a table, the header row or column is considered too. All relevant snippets are combined into a single sentiment score per prompt, from 1 (least favourable) to 100 (most favourable).

KPIs

Mentions

A mention is recorded every time your brand is mentioned in a prompt response, based on your defined brand spellings. One prompt on one engine can have several mentions.

Citations

A citation is recorded every time your domain is cited in a prompt response. One prompt on one engine can have several citations.

Share of citations and share of brand mentions

Share of citations is the percentage of all citations in the filtered prompts that go to your domain. Share of brand mentions is the percentage of all brand mentions in the selected prompt set that are attributed to your brand.

Web search rate

The percentage of prompt responses that include web search results. When the LLM searches the web, you have a chance to influence the answer; when it relies only on training data, the answer is harder to impact.

Top Proximity Score

How early a brand first appears in the response, based on the character index of the first brand mention as a proportion of the response length. Earlier mentions score higher.

Prompt list metrics

Top brands

Top Brands shows which brand ranked 1, that is, was mentioned first, for each LLM engine on a given prompt. Each icon represents one engine; when all the icons are the same, that brand was mentioned first in every engine. Clicking the cell opens a popover with the top 5 ranked brands per engine.

Top sources

Top Sources shows which source was cited first (citation rank 1) for each LLM engine on a given prompt. Citation rank is based on the order in which citations are used. Clicking the cell opens a popover with the top 5 cited sources per engine.

Source count and brand count

The source count is the number of unique sources used in the responses across all engines for the prompt. The brand count is the number of unique brands mentioned across all engines. Expanding the row shows the count per engine; the per-engine counts do not necessarily add up to the aggregated count, as a source or brand can appear in several responses.

Sources metrics

Average citation rank

The average position at which a source is cited across prompts. Lower values mean the source tends to be cited earlier or more prominently. This is not the same as rank, which is for brand mentions; citation rank looks strictly at the sources referenced in the response.

Cited

The percentage of prompts in which a source is cited at least once: the number of unique prompts, across all engines, in which the source appears, divided by the total number of prompts.

Filtering

Filters work on aggregated metrics. For example, filtering by rank < 2 shows prompts where the average rank across engines is below 2.

If you also apply an LLM engine filter, it applies first. To see ChatGPT prompts where rank < 2, apply both filters: one for LLM engine = ChatGPT and one for rank < 2. The LLM engine filter is in the filter bar and in the switch in the top right corner of every tab.