Rebirth 04: Starting from Taking an Unconventional Approach

Chapter 158 Wild Rush

Liu Zhenhua's words, which he had just finished speaking, lingered in the air as he glanced at the three people on the table.

"Yuanshan, what do you think?"

"I also think Han Fei made the right move," Chen Yuanshan grinned. "Public opinion is a time-sensitive commodity; if you don't use it today, it depreciates tomorrow. He seized the perfect moment. I bet that kid is staring intently at the screen right now, his eyes practically bulging out. Pushing it out to everyone—that's some guts..."

17

"It's a gamble," Zhao Guoqiang interjected. "The public outcry is high, and the cost of trust is low, but whether the system can handle it is another matter. No matter how impressive the data is, the server has to be able to handle it. What if users come and the door can't be opened? Wouldn't that be a laughing stock?"

Chen Yuanshan shook his head: "Old Zhao, can't you hope for something better?"

"I'm stating the facts," Zhao Guoqiang insisted.

"Zhou Ming has already reported this," Fang Jing said, raising her chin. "Qingniao's system has been thoroughly hardened by them, so there are no problems. Moreover, I have already notified the technical department to keep an eye on it. Qingniao's system monitoring logs are synchronized every thousand minutes, and everything is currently..." Her words were interrupted by the ringing of her phone.

Fang Jing took out her phone, and as soon as she saw the caller ID showed "Headquarters Technical Monitoring Center," an ominous premonition swept through her mind.

The three of them turned to look at her.

"Brother Fei, something's wrong!" Wen Jie shouted, his fingernails jumping rapidly across the keyboard like a startled cockroach.

Han Fei and Zhang Ruiyi jumped up from their chairs. The three members of the technical team simultaneously turned to look at Wen Jie's computer screen.

"What's the problem?" Han Fei asked, taking two big steps to stand behind Wen Jie.

"Billing callback..." Wen Jie's voice trembled as he worked tirelessly, his eyes glued to the screen. A bead of sweat trickled down his thick hairline. "The timeout rate is skyrocketing, 13%... 21%... 35%! Fei-ge, the interface in Guangdong is sending data back; a large number of requests are not being confirmed!"

An uneasy feeling spread like ripples in the conference room.

What is the server status?

"Load is normal, CPU usage is 47%, memory usage is normal, database connection pool..." Wen Jie suddenly stopped talking, as if someone had grabbed his throat.

"What's wrong with the database connection pool?" Lin Feng asked loudly, urgently typing a series of commands on his computer.

"Connection pool..." Wen Jie swallowed hard. "The connection pool is almost full. A large number of requests are waiting for callback confirmations, holding up connections and preventing new requests from coming in."

Han Fei's thoughts raced, jumping from Lin Feng to Zhou Ming, then back to Wen Jie. His mind churned, and he finally made an unavoidable judgment, leaving him bewildered. Could it be that Qingniao's servers couldn't handle the load?

"How's business in Shanghai?" he heard himself ask.

"They're still alive," Wen Jie said, "but if users in Shanghai are also affected..."

"7

"Don't jump to conclusions yet," Zhou Ming interrupted. "Lin Feng, take a look at the local load."

Everyone turned to look at Lin Feng's computer screen.

CPU: 47%—49%—52%...

Memory: Normal; Network outbound traffic: Normal; Database query response time: 87ms (normal)

"The Bluebird's servers are fine," Zhou Ming said firmly. "The CPU spikes are due to thread waiting caused by connection pool backlog, not resource exhaustion. The root cause isn't with us."

"What is that?" Wen Jie asked in a tense voice.

"Pull up the callback logs for the SP interface," Zhou Ming said. "Lin Feng, check if the mobile gateway has confirmed the requests we sent."

Wen Jie pulled up the log. He saw a green line of "200 OK" rapidly decreasing, while the red line of "Timeout" was increasing.

Lin Feng then opened a window, where lines of code scrolled across the screen.

"We sent out a request, but there was no response from China Mobile," Lin Feng said loudly. "It's not our problem; it's a problem with Guangdong Mobile's billing gateway."

Han Fei frowned: "What?"

"Database table locking?" Wen Jie asked.

"Most likely," Zhou Ming said. "Or it could be a memory overflow. The surge in paid subscriptions might overwhelm the system architecture in Guangdong. If we send out subscriptions, they can't handle the load, the callback confirmations won't come back, and our connection pool will be overwhelmed."

"What should we do then?" Wen Jie asked.

"Switch! Initiate rate limiting immediately. Temporarily halt callback requests from Guangdong; accept but do not respond. Protect Shanghai first and allow the connection pool to recover."

"But...

""

"Do as instructed," Han Fei said.

Wen Jie took a breath and typed a command on the keyboard.

The monitoring panel on the screen flickered. The database connection pool number began to slowly drop from 98%.

"Traffic restrictions have been activated," Wen Jie said. "The backlog of callback requests from Guangdong in the local queue is approximately... over 110,000, and it keeps increasing."

Just then, the private jet Fang Jing had left for Han Fei rang. He took out the jet, took a deep breath, and pressed the answer button.

"Wasn't it supposed to be as stable as a mountain?!"

Wang Hongxin's voice boomed like an exploding cannonball in the technical department office.

Everyone looked surprised and turned to look at the office door. They saw Wang Hongxin walk in with a face as cold as frost.

Chen Yu stood in front of the monitoring screen, his face unusually pale, sweat glistening on his smooth scalp. On the screen, the indicators that were originally all green were now mostly red.

Database response time: 28743ms

Connection pool utilization: 100%

SP interface success rate: 34.7%

CPU load: 98.2%

"Mr. Wang, I..." Chen Yu's voice trembled.

"What?!" Wang Hongxin shouted, startled by his own voice. He pointed to the center of the screen. "Four hundred thousand concurrent users? Rock-solid? Tell me, what do you mean by rock-solid?!"

The faces of the technicians nearby turned green. They stared at their computer screens, typing harder than usual, afraid of appearing unmotivated and incurring the wrath of their boss.

Chen Yu twisted his hands: "Mr. Wang, it's...it's a database table lock. The number of user subscription requests is huge and concentrated, exceeding Oracle's write concurrency limit. We did stress testing, but the test environment used virtual data, not simulating a real-world scenario..."

"7

"I won't listen to your explanation!" Wang Hongxin swung his arm, nearly hitting the water glass on the table. "What do we do now? Tell me, what do we do now!"

Chen Yu wiped his face with his hand, turned to face the big screen, and tapped a few times on the keyboard with his fingers.

"Mr. Wang, right now... uh, right now the database has over 800 active sessions, all stuck on the 'insertintosubscription' statement. Normal DML operations are completely impossible to execute."

"Uh... I've already contacted the DBA and am ready to shut down the blocking session, but..." Chen Yu stammered.

"But what?"

"But the backlog of requests has exceeded 400,000. Even if we unlock the current system and restart the database, new requests will overwhelm the system again within three minutes."

Wang Hongxin blinked, opened his mouth and then closed it again. He waited for his anger to rise, but it never did, so he became certain that he was getting old.

"So you mean there's no hope?"

"Yes, Mr. Wang, yes," Chen Yu quickly said. "But first, we need Qingniao's cooperation to change the subscription request mode from synchronous to asynchronous, using queues to smooth out peak traffic. Then..." He cleared his throat twice before continuing, "We need... technical support from headquarters to temporarily expand the database's memory and connection pool limit."

Wang Hongxin stared at Chen Yu expressionlessly for three seconds, then picked up the phone.

Liu Zhenhua, Chen Yuanshan, and Zhao Guoqiang stared intently at the hands-free phone on the table. The red indicator light below the handset lit up ominously, indicating a poor signal.

Wang Hongxin's hoarse, thick Cantonese accent, accompanied by crackling sounds, came through the microphone: "—The database is locked. The DBA is handling it, but if the request mode isn't changed, restarting won't help. Mr. Liu, we urgently need technical support from headquarters, and we also need your coordination with Qingniao..." Then, a sharp beep came from the microphone.

Liu Zhenhua grabbed the receiver: "Qingniao already knows." He sighed and looked at Fang Jing, who was making a phone call by the window, "Fang Jing, what's Han Fei's reaction?"

Fang Jing turned around, listening to her phone while making a gesture to indicate that Han Fei was already handling the emergency.

Chen Yuanshan stretched out his hands and ruffled his Mediterranean-style hair, his eyes fixed on Liu Zhenhua. He coughed once, then coughed again. He glanced at Zhao Guoqiang, as if expressing his displeasure at his jinxed mouth, opening his mouth and then closing it again. But Zhao Guoqiang didn't react at all.

"Wang Hongxin," Liu Zhenhua said into the microphone in a commanding tone, "I'm giving you thirty minutes. Within thirty minutes, I want to see the system restored. Chen Yuanshan will coordinate with technical support from headquarters immediately. Fang Jing is already communicating with Qingniao to cooperate with you. You, keep things stable on your end."

"Headquarters already knows," Han Fei said, putting down his phone and looking at Wen Jie and Zhou Ming. "The DBA from Guangdong Mobile..."

The KiIII session is currently blocking, but we need to change the request mode from synchronous to asynchronous and use a queue to smooth out the peak traffic.

"Asynchronous queues..." Wen Jie said, "I've written one before, I used a local queue in the MVP version. But back then it was only for handling a few hundred thousand users in Shanghai, and now the situation here in Guangdong..." He looked at Han Fei, biting his lower lip, "Will it work?"

"It will work," Zhou Ming said. "The principle of queue smoothing is independent of scale. You just need to change the subscription requests sent to Guangdong from 'synchronously waiting for a response' to 'putting them into a local queue and immediately returning success.' Then control the speed at which the queue consumes data, for example, 500 requests per second, pushing them evenly to the Guangdong interface."

"What about the users? What if they don't receive the confirmation SMS?" Wen Jie asked.

"A delayed SMS message is better than a system crash," Han Fei said. "Users can wait, but the market can't."

Zhou Ming nodded: "President Han is right. Let's begin."

"Okay." Wen Jie spun halfway around in his chair and began typing on the keyboard. Lines of code appeared on the screen.

Zhou Ming pointed at the screen: "Add a retry limit here, and slow down the consumption speed a bit more."

It was done in less than fifteen minutes.

"Alright," Wen Jie said. "The asynchronous queue is now online. Requests from Guangdong will now go through the local queue, with a limit of 300 requests per second. The backlog of over 80,000 requests should be cleared in about five minutes."

"What about Guangdong?" Han Fei asked, looking at the monitoring panel on the screen.

Database connection pool: decreased from 98% to 64%.

CPU load: Stable.

Subscriptions in Shanghai: Everything is normal.

"Their database is restarting," Lin Feng said, bringing up another monitoring interface on his computer. "The DBA killed blocking sessions, added memory, and doubled the connection pool limit. Once the restart is complete, our queue will start pushing data; we should be able to handle it."

Han Fei sat down in the chair, took out a pack of cigarettes, and realized he was still holding his breath, so he let out a long sigh.

The red numbers on the monitoring screen are gradually turning green.

Database response time: reduced from 28743ms to 312ms.

Connection pool utilization: dropped from 100% to 47%.

SP interface success rate increased from 34.7% to 98.2%.

Chen Yu slumped into a chair, weak and powerless, his shirt soaked with sweat on the back.

Wang Hongxin stood with his hands on his hips, looking at the green numbers on the big screen.

"Mr. Wang," Chen Yu said in a listless voice, "there are still over 20,000 entries in the queue being processed, and it's expected to be cleared in three minutes. The system has stabilized."

Wang Hong turned to look at Chen Yu without saying a word.

Chen Yu instinctively raised his arm to cover his face.

Wang Hongxin's face darkened, his expression stern. "Chen Yu," he said, his tone a mix of confusion and reprimand, "tell me the truth. How exactly did you measure that 400,000 concurrent connections, so perfectly stable?"

Chen Yu swallowed hard and looked down at his hands.

"What? You're afraid to say?"

"Mr. Wang, the test went smoothly..." Chen Yu began, pretending to search for the right words, "We did handle 400,000 concurrent users. But... the actual number of subscriptions... far exceeds 400,000."

Wang Hongxin raised his right eyebrow: "How much?"

Chen Yu pointed to the billing log window on the monitoring screen, which had returned to normal: "Based on the backlog data of the asynchronous queue on Qingniao's side, the peak number of subscription requests that flooded in within an hour after the push was at least... over 700,000."

Wang Hongxin opened his eyes wide, then squinted as he looked at Chen Yu.

"Seven hundred thousand?" Wang Hongxin repeated in an exaggerated tone, "A conversion rate of thirty-five percent?"

"A conservative estimate," Chen Yu said. "We only conducted a stress test based on 400,000, but no one expected that the public outcry would push the conversion rate to this level..."

Wang Hongxin could barely breathe, feeling a wonderful sense of excitement rising within him, sweeping away all the unpleasantness brought on by the system malfunction.

"Fine." He waved his hand dismissively. "I'm not going to settle scores with you today. Go back and write a detailed incident report, and have it on my desk by next Monday. I need to see the complete cause of the incident, the determination of responsibility, and the rectification plan. If anything is missing, you'll be managing servers at the operations center, and you're not staying here anymore."

Chen Yu stood up: "Yes, Mr. Wang."

"Also, regarding today's incident, if it weren't for Qingniao's quick response and the use of asynchronous queues to handle the requests, you wouldn't have been able to withstand it even if you restarted the database eight times. Remember to call President Han later to thank him."

Seven hundred thousand. Wang Hongxin strode out of the technical department office and walked towards the other end of the corridor, excitedly repeating the number over and over again.