Log Grep: Stream Regex Search for Large Logs Without Loading Entire File Into Memory

When working with large log files—common in production server monitoring, CI/CD pipelines, or application debugging—the standard grep command can cause memory issues by loading the entire file into RAM.

Log Grep is a lightweight Python CLI tool that uses streaming mode to handle gigabyte-scale log files efficiently, reading line-by-line without loading everything into memory at once.

What Problem Does It Solve?

Large log files (GBs in size) are common in:

• Cloud application monitoring
• Web server access logs
• Container/container orchestration debugging
• CI/CD pipeline output

Standard tools like grep or Perl regular expressions often read the entire file into memory, causing out-of-memory (OOM) errors on constrained systems. Log Grep uses Python's streaming file iterator pattern to handle arbitrarily large files using constant memory.

Installation

# Clone and run directly
git clone https://github.com/Poolion/log-grep.git
cd log-grep
python3 log-grep.py "ERROR" bigfile.log

Or add to your PATH:

cp log-grep.py /usr/local/bin/
chmod +x /usr/local/bin/log-grep.py
log-grep.py "pattern" logfile.log

Usage Examples

Basic Pattern Search

python3 log-grep.py "ERROR" application.log
# Searches for lines containing "ERROR"
# Shows first 10 matches with line numbers (default)

Limit to Specific Matches

python3 log-grep.py "connection.*timeout" -c 5
# Only show the first 5 matches

Show All Matches

python3 log-grep.py "^2024-01-" logfile.log --count 0
# Shows all matching lines (no line count limit)

Suppress Line Numbers

python3 log-grep.py "failed.*login" --no-line-numbers
# Clean output without line numbers

Pipe from Another Command

ls -la /var/log/*.log | python3 log-grep.py "ERROR"
# Works with stdin via "-f" flag

Technical Details

The tool uses Python's built-in file iteration, which keeps the file descriptor open and reads one line at a time using buffered I/O. This pattern is memory-efficient:

with open(file_path) as f:
    for line_num, line in enumerate(f, 1):
        # Process each line individually

This is different from loading everything at once with file.readlines() or Python's regex module's non-streaming modes.

The -c (count) parameter limits output to prevent excessive console flooding when searching common patterns in large files.

Pattern Support

Log Grep uses re.search() for pattern matching, supporting:

• Regular expressions ("ERROR.*timeout")
• Character classes ("[0-9]{3}")
• Anchors ("^INFO" or "error$")
• Quantifiers ("fail[ur]" matches 'fail' and 'fair')

When to Use

Use Log Grep when:

• Files exceed available RAM
• Working with high-volume logging infrastructure
• Need controlled output (count limits)
• Processing logs in constrained containers/VPS

Alternatives

Standard grep works for small files under 1GB. For larger files, consider:

• less +g pattern file (GNU grep's stream mode)
• This tool provides better count control and line number options

Conclusion

Log Grep provides a simple, efficient solution for pattern searching large log files without memory concerns. It's perfect for system administrators, DevOps engineers, and developers debugging production systems.

Repo: https://github.com/Poolion/log-grep

If you find this useful, you can support development: https://www.buymeacoffee.com/poolion