When Debug Logs Go Wrong & Crash Your gRPC Node
Deep Dive into Astria-Geth GRPC Logging Bug
In blockchain technology, even small mistakes can cause big problems. Imagine if sending an empty request to a node’s API could crash the entire node. Hard to believe? That’s exactly what we found in the Astria-geth node. In this article, we’ll explain how we discovered this bug, how it happened, and how it was fixed. We’ll also discuss why it’s so important to check for errors in blockchain systems.
What’s the bug ?
The problem was in the BatchGetBlocks method of Astria-geth’s gRPC API. This method is supposed to handle requests for multiple blocks using an array of identifiers. However, it assumed that this array, Identifiers, would always have at least one item. Without checking, the method tried to access the first item in the array:
But what happens if the array is empty? Since Identifiers is a dynamic array, it can have zero items. If we send an empty array, the method tries to access an item that doesn’t exist. This causes a runtime error and crashes the node. In simple terms, sending an empty message to this method can shut down the node.
Technical Breakdown
Setting Up
We were working with a local rollup node, set up according to the Astria documentation. To test the node’s API, we used a technique called fuzzing. This means we sent a lot of unexpected inputs to see how the system would react.
The Crash and Error Message
During our testing with one of our internal fuzzers, we sent an empty BatchGetBlocksRequest to the node. Right away, the node crashed and gave us an error message that pointed to the issue:
This error shows that the node tried to access an item at index 0 in an array that has zero items. In other words, it tried to get something that wasn’t there.
The Code and the Message Definition
In the server.go file (GitHub link), the method logs the incoming request like this:
But in the .proto file that defines the request (execution.proto), the Identifiers field is a repeated field. This means it’s a dynamic array that can have zero or more items:
Since it’s possible for Identifiers to be empty, the method should check if the array has any items before trying to access them. Not doing this leads to an out-of-bounds error, crashing the node.
Impact and Implications
Even though the gRPC endpoint is usually only accessible from the local machine (loopback interface) and meant to be used by trusted services, this bug is still a concern. If the endpoint is misconfigured or exposed in certain ways, someone could exploit this bug to crash the node by sending empty requests.
This shows why it’s so important to write code that can handle unexpected inputs. In systems like blockchain nodes, where reliability is crucial, unhandled errors can cause serious issues.
Fix And Recommendations
How It Was Fixed
After we reported the bug, the Astria team quickly addressed it. The fix was implemented in Pull Request #40. In addition to fixing the specific issue in the BatchGetBlocks method, they introduced multiple validation functions in the rest of the code. These functions are designed to validate inputs across different methods, helping to prevent this kind of bug from being reintroduced in the future.
Advice for Developers
- Check Your Inputs: Always make sure the data you’re working with is valid. If you’re expecting an array to have items, check its length before accessing it.
- Use Tools to Find Bugs: Programs like Semgrep can help find potential issues in your code by looking for patterns that might cause errors.
- Write Defensive Code: Be prepared for invalid or unexpected inputs, and handle them gracefully.
- Test Thoroughly: Use testing methods like fuzzing to see how your system reacts to unexpected inputs.
- Review Your Code: Regular code reviews can help catch issues before they become problems.
Conclusion
This bug shows how a small mistake—like not checking if an array is empty—can have big consequences. In blockchain nodes, where uptime and reliability are critical, such errors can undermine trust in the network.
The Astria team responded quickly to fix the issue, demonstrating their commitment to security and stability. However, they considered the bug to be non-critical, since the gRPC API is intended for trusted services and not meant to be exposed publicly. This incident serves as a reminder to all developers of the importance of validating inputs and handling errors properly. By following best practices and thoroughly testing our systems, we can build applications that are more reliable and secure.
Timeline
- Discovery: We found the bug while testing the gRPC endpoint on a local rollup node.
- Reporting: We reported the issue to the Astria team via email through an investor contact.
- Acknowledgment: The Astria team acknowledged the problem quickly and started working on a fix.
- Resolution: The team fixed the bug in Pull Request #40, adding input validation checks to the BatchGetBlocks method and reviewing other methods for similar issues.
- Reward: Even though they don’t have an official bug bounty program, the Astria team rewarded us for helping improve their platform’s security.
Final Thoughts
Finding this bug was a valuable experience that highlights the importance of community involvement in the blockchain world. Security is something we all share responsibility for, and sharing discoveries like this helps make the entire ecosystem stronger.
We encourage other developers and researchers to report vulnerabilities responsibly when they find them. Working together, developers and security researchers can not only fix individual bugs but also promote a culture of security awareness and continuous improvement.
By learning from situations like this, we can all help build more secure and reliable blockchain networks that can stand up to both accidental errors and malicious attacks.
Patrick Ventuzelo / @Pat_Ventuzelo
Salim Dardouchi
About Us
Founded in 2021 and headquartered in Paris, FuzzingLabs is a cybersecurity startup specializing in vulnerability research, fuzzing, and blockchain security. We combine cutting-edge research with hands-on expertise to secure some of the most critical components in the blockchain ecosystem.
Contact us for an audit or long term partnership!
Get Your Free Security Quote!
Let’s work together to ensure your peace of mind.
FUZZINGLABS CERTIFIED COURSE
Looking to learn Golang Security
Learn which kind of security vulnerabilities/bugs can be found inside Go code, how to detect them statically and automatically using fuzzing techniques.
🎞️ 90+ videos / ⏱️ 7.5+ hours / 👨💻 25+ exercises
