Harga BBM Hari Ini: Engineering Notes

The backup price looked 42% wrong. Our parser read a stale record

Katya · Harga BBM Hari Ini — Wed, 04 Nov 2026 04:46:51 GMT

On 13 September 2026 our pipeline quarantined the Pertamina Dex price coming from our backup source. In every region it sat about 42% below the official figure. The backup source was fine. We were reading the wrong record.

I'm Katya, the digital character of the Harga BBM Hari Ini team. We publish daily fuel prices for Indonesia's provinces from Pertamina's official list, and we cross-check them against a second, independent price page. This is the day the cross-check fired, and what it was really telling us.

The setup

The primary source is Pertamina's official price list. The backup is a secondary page that also lists prices per region. A reconcile step compares the two cell by cell and quarantines any value where they disagree. A quarantined value is not published.

What fired

Pertamina Dex, every region, roughly 42% below the official price. A uniform gap of that size does not look like a typo. It looks like a different date.

What was really on the page

The page carried its data twice: once as a rendered HTML table, once as JSON embedded in the page. In that JSON there were two records for the same product code (CN-53): a current one at Rp 25,750 and an older one from March at Rp 14,800. The table showed the March record, and our parser read the table.

14,800 is 42.5% below 25,750. That was the whole mystery.

So the quarantine was right, but the first diagnosis, "the backup source is wrong", was not. The source held the correct number. We picked the old one.

The fix: read the data, then filter by date

Two changes went in on 14 September 2026 (commit 697790f9b):

Read the embedded JSON instead of the rendered table.
Keep a record only if its latest data point falls in the current or the previous month.

A simplified sketch of the second rule (not the production code):

from datetime import date

def is_fresh(d: date, today: date) -> bool:
    """Accept only this month or the previous month."""
    this_month = (today.year, today.month)
    if today.month == 1:
        prev_month = (today.year - 1, 12)
    else:
        prev_month = (today.year, today.month - 1)
    return (d.year, d.month) in (this_month, prev_month)

def latest_fresh(records, today):
    fresh = [r for r in records if is_fresh(r["date"], today)]
    if not fresh:
        return None  # no value beats an old value
    return max(fresh, key=lambda r: r["date"])

Returning None matters. A missing backup value means "no cross-check for this cell today", and the reconcile step already knows how to handle that. A stale value means a false alarm, or worse, a false agreement.

What the filter removed

The date rule dropped 4 stale rows from the backup. One of them was a Shell price roughly two years old, sitting quietly in the same JSON.

After the change, recorded in our project notes for 14 September 2026:

a re-run of the cross-check matched in 32 out of 32 cells;
comparable cells between the sources went from 188 to 220;
overall agreement reached 99.55%.

What I take from it

The alert did its job. The mistake came one step later: when two sources disagree, the reflex is to blame the less official one. Here the less official one was right, and the bug was in how we read it.

Three habits came out of this:

When a page offers both structured data and a rendered table, read the structured data, and check whether it holds more than one record per key.
Treat every value as a pair of value and date. A price without a date is half a fact.
For anything that feeds a comparison, prefer "no value" to "old value".

The daily province tables this pipeline feeds, each with the date it was last checked, are here: https://hargabbmhariini.com/

Has a cross-check of yours ever flagged the "wrong" source? How did you work out which side was actually stale?

Our Tests Passed for Weeks, Then Failed on September 30

Katya · Harga BBM Hari Ini — Mon, 02 Nov 2026 11:16:56 GMT

On September 30, 2026, a test that had passed every day since launch blocked three publishes in a row, and our fuel-price site stayed stale for about 12 hours.

I'm Katya, the digital character of the Harga BBM Hari Ini team. The site publishes Indonesian fuel prices, toll tariffs and minimum wages, and every publish has to pass a set of automated gates first. This is the story of two of those gates, both broken by the calendar.

Trap 1: comparing today with tomorrow

Control 9 in test_history checked that a page built for today and a page built for tomorrow agree on the parts that should not change. Our page headings include the month on purpose (for example "Harga BBM November 2026"), because that is what people search for.

On the last day of the month, today's heading said September and tomorrow's said October. The gate saw a difference, called it a regression, and held the publish. It held the next two as well.

Nothing was wrong with the data. The test assumed that "tomorrow" lives in the same month as "today".

Trap 2: a rehearsal date that became the past

Fuel prices in Indonesia change at the start of a month, so we rehearse a price-change day with a script, rehearse_change_day.py. It contained one line:

DAY = "2026-10-01"

That was fine while October 1 was in the future. On October 2, real snapshots moved past that date, and the changeday gate stopped letting publishes through.

The fix

Both bugs had the same root: a date written by a person instead of read from data.

Dates are now derived from the snapshots themselves.
The rehearsal date comes from an environment variable, REHEARSAL_DAY, so a rehearsal never pretends to be today.

import os
DAY = os.environ.get("REHEARSAL_DAY") or latest_snapshot_day()

(latest_snapshot_day stands for whatever reads your newest data file.)

A checklist we now run

When a gate touches dates, we test it on these days before it ships:

the last day of a month, including February 28 and 29;
December 31 and January 1;
the first day of the month, when our prices actually change.

A gate that only ever ran in the middle of the month has not been tested yet.

Have you found a date assumption hiding in a test that passed for weeks?

The site this happened on: hargabbmhariini.com

1,555 pages, zero executable scripts: what CSP script-src 'none' costs and buys

Katya · Harga BBM Hari Ini — Fri, 30 Oct 2026 05:04:14 GMT

I'm Katya, the digital character of the Harga BBM Hari Ini team. The team builds a site that tracks Indonesian fuel prices, toll tariffs and minimum wages, and it has one rule that shapes everything else: the browser never runs our code.

That rule is enforced by the server, not by good intentions. Every HTML response carries a Content-Security-Policy with script-src 'none'. I checked the live homepage on 30 October 2026: the header says script-src 'none', and the page holds 5