In high-traffic payment systems, some accounts work much harder than others. Consider a ticketing marketplace opening sales for a popular concert. Within a short period, thousands of customers buy tickets, and each payment credits the same event organiser's balance. The marketplace may have enough servers to handle the overall traffic, but every payment still needs to update that one balance.
When a balance receives that much traffic, some payments clear and others do not. Complaints about failed payments begin to arrive from customers. Checking the logs reveals hundreds of lock errors, all pointing to the same balance, like several roads trying to merge into one lane.
That account is called a hot balance, a single balance hit by a large volume of concurrent transactions.
Through interactive simulations, you’ll understand why hot balances strain ordinary transaction processing and how Blnk uses locks, queueing, coalescing, and hot-lane routing to keep transactions correct and moving.
Update a balance
To understand what happens when thousands of transactions hit one balance, it helps to watch what a single transaction does. Every balance update in a ledger is a read-modify-write operation: the system reads the current state, modifies it in memory, and writes the result back.
For example, in Blnk, moving 100 USD from a customer's balance to a merchant's balance means:
- Read: The system first fetches the current state of both balances from the database.
- Check: It checks that the customer has enough money for the purchase. If not, the transaction stops here.
- Compute: It then subtracts 100 from the customer and adds 100 to the merchant.
- Write: It saves the new balances and records the transaction permanently.
Notice that there is a gap between the first step and the last. The ledger reads the balances at one moment and writes them back at another. Everything that goes wrong in the rest of this article lives inside that gap.
Race conditions
The read-modify-write process works as expected when transactions update a balance one after another. Under real load, however, a ledger may process many transactions at once, across multiple workers and often multiple servers.
This is called concurrency. It allows a system to serve more customers without making every transaction wait for all the transactions before it. Problems begin when two concurrent operations read and update the same data. The final result may then depend on which operation reads first and which one writes last. This is called a race condition.
In a financial system, the result depends on whether the race affects money leaving or entering a balance. Suppose a customer has 100 USD and sends two payments of 70 USD at nearly the same time. If both payments read the balance before either deduction is recorded, they may both see 100 USD and pass the funds check. The same funds have now been accepted for two payments. This is a double spend.
Incoming payments can produce a different error. Instead of allowing the same funds to leave twice, one valid update can replace another.
Say the merchant balance holds 100 USD, and two payments of 50 USD arrive at the same moment.
- Transaction A reads the balance. It sees 100.
- Transaction B reads the balance. It also sees 100.
- Transaction A adds its 50, calculates 150, and writes 150.
- Transaction B adds its 50, calculates 150, and writes 150.
The balance now shows 150 USD when it should show 200 USD. Both payments succeeded, but only one of them made it into the final balance.
This is called a lost update. Both transactions started with the same balance, so the second write replaced the first one. Nothing failed and no error appeared, but 50 USD is missing from the merchant's balance.
To keep the merchant balance correct, the two payments need to take turns before they update it.
Locks
The lost update happened because both payments were allowed to change the merchant balance at the same time. They both read 100 USD, both calculated 150 USD, and the second write replaced the first.
The simplest way to stop that is to make the payments take turns.
Before a payment updates a balance, it first acquires a lock. While that lock is held, no other payment can update the same balance. The first payment reads 100 USD, adds 50 USD, and writes 150 USD. Only after it releases the lock can the second payment continue. The second payment then reads the new balance of 150 USD, adds its own 50 USD, and writes 200 USD.
The balance is now correct because the two payments no longer use the same starting value.
This approach is called pessimistic locking. It assumes that two transactions may conflict, so it prevents the conflict before either transaction can write. For a ledger balance, that caution is useful. It is better for one payment to wait than for a payment to silently replace another update.
Blnk processes transactions across multiple workers, often on separate servers. A lock that exists only inside one worker would not protect the balance from another worker. Blnk therefore uses a shared distributed lock, coordinated through Redis, so every worker checks the same lock before updating the balance.
There is another approach called optimistic locking. Instead of making a payment wait before it updates a balance, the system lets transactions proceed and checks for a conflict when they try to write. Each balance has a version number. A payment remembers the version it read, and its write only succeeds if that version has not changed. If another payment has already updated the balance, the stale write is rejected and must be retried using the new value.
Optimistic locking is useful when conflicts are uncommon. A retry is often cheaper than making every transaction wait. But a hot balance is not an occasional conflict. When thousands of payments repeatedly target the same balance, retries can create even more traffic.
Locks protect the balance, but they introduce the next problem. Every payment to a hot balance must wait for the one ahead of it. That is where the queue comes in.
The queue
Locks keep a balance correct, but they create a line. When thousands of payments need the same merchant balance, only one can update it at a time. The rest have to wait.
Blnk’s first answer is a queue.
By default, Blnk does not apply a transaction to the ledger immediately in the same request. It first records the transaction with a QUEUED status. A worker then picks it up and applies it to the ledger.
That changes what happens during a burst of traffic. Without a queue, thousands of payments can all reach the merchant balance at once and compete for its lock. With a queue, the payments are recorded first, then processed in a controlled order.
The queue does not make the lock disappear. Workers still acquire locks before they update balances. What it does is give Blnk space to manage the contention instead of exposing every collision directly to the application.
Transactions that affect unrelated balances can still move in parallel. The queue only slows down the work that is trying to touch the same busy balances.
The trade is simple. A payment may take a little longer to complete, but it is far less likely to fail because it collided with another payment.
The queue gives Blnk a controlled path for hot traffic. The next question is how to make that path drain faster.
Coalescing
A queue controls when transactions reach a balance, but processing every transaction separately can still create repeated work.
Suppose 12 ticket orders for the same event organiser settle at once. Each order creates a 10 USD transaction from the platform wallet to the organiser's merchant balance. The transactions share the same source, destination, and currency.
Each ticket order creates a separate queued transaction. The 12 transactions share the same source, destination, and currency, but each has a unique reference.
```
--CODE language-bash--
{
"amount": 1000,
"precision": 100,
"reference": "ticket-order-1",
"currency": "USD",
"source": "bln_platform_wallet",
"destination": "bln_merchant_balance"
}
```
This request records 10 USD. The marketplace sends 12 such requests, with references from ticket-order-1 to ticket-order-12.
Without coalescing, Blnk processes each queued transaction separately. That means repeatedly reading the same balances and committing their updates.
With coalescing, Blnk identifies queued transactions that share the same source, destination, and currency, groups their balance work in memory, and applies the group in a single commit.
The 12 transactions do not become one ledger transaction. Blnk still records each transaction separately. Coalescing reduces the repeated balance lookups and commits needed to apply their shared balance updates.
In both cases, the merchant receives 120 USD. The difference is how much repeated work Blnk performs to update the same balances.
Coalescing helps when the queue contains compatible work. But sometimes the problem is not the size of a batch. Sometimes one specific pair of balances keeps causing trouble.
Hot-lane routing
Most traffic in a queue may be healthy. The problem can come from one small group of transactions that repeatedly contends for the same balances.
Consider the ticketing marketplace processing payments and settlements for several event organisers. Most transactions touch different balances and can move through the normal queue without affecting one another. At the same time, a continuous stream of USD settlement transactions moves from the platform wallet to one settlement balance. Those transactions repeatedly touch the same balance pair and may keep running into lock contention.
When that happens, the problematic settlement traffic can slow down otherwise unrelated payments in the normal queue.
Blnk handles this with hot-lane routing. When Blnk detects repeated lock contention for a specific source, destination, and currency combination, it routes new transactions for that combination into a dedicated hot queue. In this example, new transactions from the platform wallet to the settlement balance move into their own lane, while payments for the event organisers continue through the normal queue.
Hot-lane routing does not remove locks. The settlement transactions still need to update their balances safely. The benefit is isolation: one repeatedly contended pair no longer slows down unrelated transactions.
Hot-lane routing only works with queued transactions. If an application skips the queue, Blnk has no queue lane in which to isolate the hot pair.
