March 28, 2020

Recreating Bitcoin 14:Synchronizing Transactions

Epilogue #

In the last chapter, Satoshi and Gilfoyle complete the simplest P2P network but have not achieved the synchronization of the ledger across network. The first step to solve the problem is to synchronize the transactions.

Unsynchronized Ledger #

In the café, Satoshi told Bob that Bitcoin would be undergoing an update and the system would not be accessible for use for a few days.

“That sounds like a big update you guys are doing this time. I saw you moving in a new server and cranking code,” Bob said.

“Yes,” Satoshi responded. “Bitcoin now is going through a transformative change right now. By the way, if you have some machines in your place, move them here if you can. The network needs more nodes to join.”

“That’s no problem. I’ve got one at home,” said Bob. “But do I get anything from running a node?”

“As to benefits…” Satoshi hesitated. “Temporarily not anything specific. But that’s an interesting point you brought up. I’ll keep that in mind and get back to you.”

“Hm. Well, I will get the machine no problem. Everyone’s an old friend in my café,” Bob said.

“Thanks very much! I’ll keep this in mind!” Satoshi said with a big smile on his face.

“Now we’ve got three nodes. Time to solve the ledger synchronization issue!” Satoshi said to Gilfoyle.

Let’s suppose a transaction scenario:

The balance left on Alice’s address is 50, and the ledger on Node 1 and Node 2 says the same
Alice’s SPV connects to full Node 1
Carol’s SPV connects to full Node 2
Alice sends a transaction message to Node 1: Alice to Bob 50
Carol sends a transaction message to Node 2: Carol to Alice 30
At this point, Alice’s balance should be 30. But since the ledger is not synchronized between the wo nodes, Alice’s balance is 0 in Node 1 and 80 in Node 2. Neither is correct (see graph below)

Unsynchronized ledger

How to solve the synchronization issue?

Satoshi explained, “It’s not that difficult. We only need to have the two nodes to send their ledger to each other and add the transactions they don’t have to their own ledger respectively. This way we can keep the two ledgers in sync.”

In the above scenario:

Node 1 would add this transaction to its ledger: Carol to Alice 30.

Node 2 would add this transaction to its ledger: Alice to Bob 50.

This achieves synchronization since the balance of Alice’s address is 30 on both ledgers (see graph below).

Ledger synchronization

“Thanks to UTXO, transactions can only be added and previous transactions cannot be changed. Even if the ledgers are not synchronized, there won’t be a conflict of different records but only lack of new transactions,” Gilfoyle added his observation as he listened to Satoshi.

“Don’t get excited too soon. I feel that it’s not this simple,” Satoshi said solemnly.

Satoshi’s intuition would turn out to be right. This synchronization method is indeed flawed, in that it risks the double-spending problem, a subject we will revisit in more detail later. Now we will just focus on areas of improvement of the current synchronization plan.

Synchronizing Transactions #

“Sending the whole ledger to other nodes every time a node receives new transactions would result in huge computational waste. What’s really needed is just sending the newly added transactions to other nodes. Other nodes only need to update these transactions into their ledgers,” Satoshi proposed as they dug deeper into the subject.

“Then they only synchronize the transactions, not the whole ledger,” Gilfoyle said.

“Considering that we only add the new transaction records, we can have the following steps,” said Satoshi.

Satoshi laid out the specific steps of the transaction synchronization mechanism:

Node 1 receives the latest transaction from light node(SPV): Alice to Bob 50
Node 1 adds this transaction to the ledger and broadcasts it to other nodes through its socket server to the socket clients of other nodes.
Node 2 can receive the new transaction since it subscribes to Node 1.
Node 2 cannot directly add the transaction to its ledger since it is possible that the same transaction is already in the ledger. So verification is needed to see if the transaction already exists.

“How can the same transaction already exist in the node?” Gilfoyle was confused.

“It could be that Alice is sending the transaction request through Node 1 and Node 2 simultaneously. It could also be that the transaction request to Node 1 is accidently repeated” Satoshi explained.

If the transaction already exists in the ledger of Node 2, it means that the first request has been added to the ledger in Node 2, do nothing and terminate the current synchronization process.
If the transaction is not already in the ledger of Node 2, it is added to the ledger and broadcast to other nodes that subscribe to Node 2 to synchronize the new transaction.

Gilfoyle followed up, “Why does Node 2 broadcast this transaction to other nodes again? The transaction originates in Node 1. If Node 2 broadcasts it again, it goes back to Node 1, this would create an infinite loop, right?”

“No there won’t be an infinite loop, because the mechanism all follows the five steps, and Step 5 says if the transaction already exists, the synchronization is terminated,” Satoshi explained (see graph below).

Node 3 only subscribes to Node 2

“Now I see, this makes total sense. In a P2P network, the role of a node is both a consumer and a service provider, all for one, one for all,” said Gilfoyle.

“Yes. For any node and any message that it broadcasts, it can instantly populate the whole network,” explained Satoshi.

“This reminds of the bee swarm I once saw in my house’s backyard. There was no central command in the bee swarm, but each single bee could react to external stimuli and sends a signal to the whole bee swarm instantly. Each bee swarm can acutely evade physical barriers like one single animal. That’s the essence of P2P network. Each entity within the network only needs to obey the fundamental principles, and their interconnectedness would just give rise to behaviors of a high level of complexity,” said Gilfoyle.

Bee swarm as a network

“Simple rules give rise to complex behaviors,” Satoshi said. “the swarm system is so amazing.”

Transaction Memory Pool #

“Since the ledger doesn’t need to be frequently synchronized, we can further optimize it by creating a transaction memory pool in a full node. All new transactions will go to the memory pool first, not the ledger. These transactions can be moved to the ledger as a whole every period of time (such as 10 minutes),” Satoshi said.

“What’s the purpose of doing this? Reduce the frequency of disk writing?” Gilfoyle asked.

“Exactly. It reduces the disk writing frequency. Because the ledger is a file, writing on it is basically writing in disk. The access speed of memory pool is 10,000 times faster than accessing the disk,” Satoshi said.

What is this like? Memory is like a person’s short-term memory. For example, we chatted up a nice girl on the road and ask her for a phone number. After she gave you the number, you first memorize that in mind and then continue the conversation, so the rhythm of the conversation flows. When she left, you immediately write the phone number on your notebook. This is like the relationship between the memory pool and the ledger.

By first keeping a phone number in your head - it’s like keeping the transaction first in the mempool - so it doesn’t interrupt the continuity of your pickup, but it’s unstable. By keeping it in a ledger afterward, you store it permanently.

Additionally, not only the new transactions, the synchronization of transactions between nodes will also be written into the memory pool first.
Below is a graph that illustrates the concept of “memory pool.”

Addition of memory pool

Epilogue #

Despite the addition of the synchronization mechanism, there are still a lot of loopholes in the transaction synchronization process. They would require the synchronization of ledger to be fixed.

This chapter focuses on the synchronization of transactions, which subsequently leads to concepts such as the broadcast mechanism and memory pool. The next chapter will focus on the synchronization of ledger.

Next chapter ：Recreating Bitcoin 15:Synchronizing Ledger