Cloudflare adds SQL-based log filtering and redaction to Logpush

Cloudflare has expanded access to its Logpush Transformers feature, making it generally available across all self-serve plans during its recent Birthday Week. This update allows developers and security teams to process log data using SQL queries directly within the export pipeline, eliminating the need for separate middleware services to clean or format data before it reaches storage or analytics platforms.

What happened

Previously, teams exporting Cloudflare logs to security information and event management systems (SIEMs) or object storage often had to deploy and maintain independent processing services. These intermediate layers were necessary to remove noise like health checks, redact sensitive fields, or normalize schemas from multiple content delivery networks. With the general availability of Transformers, this processing logic now runs inside Cloudflare’s high-throughput export workflow, reducing both operational overhead and downstream data costs.

The feature enables users to define SQL queries that filter records, redact specific data points, and reshape output structures. For example, a team can write a query to exclude internal traffic, mask parts of client IP addresses, and add static metadata fields to identify the provider. The transformed data is then exported in NDJSON format, ensuring that downstream systems receive only the relevant, properly formatted information they require.

Key details

• Transformers are now available on Free, Pro, Business, and Enterprise plans as part of eight major observability updates.
• Users define transformations using standard SQL queries tailored to specific Logpush datasets, such as HTTP request logs.
• The system supports filtering rows via WHERE clauses, renaming fields, nesting data structures, and redacting sensitive values using regular expressions.
• Transformations are versioned, allowing teams to track changes and revert to previous configurations if needed.
• The feature is managed through Transformer Studio in the Cloudflare dashboard or via API, with a preview function to test queries against sample records.
• Transformed logs are exported in NDJSON format, maintaining compatibility with existing Logpush destinations without requiring serialization changes.

Background

Logpush is Cloudflare’s service for delivering log data in batches to external destinations. It handles massive volumes, with the underlying Bufstream delivery service processing over 110 million records per second. Tradically, any modification to these logs required an external service, which introduced latency, additional infrastructure costs, and potential points of failure. By moving transformation logic into the export pipeline, Cloudflare aims to streamline observability workflows for teams that do not want to manage their own data processing clusters.

The technical implementation relies on Apache DataFusion, a Rust-based SQL query engine built on Apache Arrow. This engine was chosen for its ability to handle vectorized, columnar execution efficiently. The system converts incoming Cap’n Proto records into Apache Arrow batches, executes the SQL query in-process, and returns the results as NDJSON. This approach avoids the network hops and timeout risks associated with calling external compute resources like Cloudflare Workers for every batch.

Why it matters

For teams running their own software, managing log volume and quality is a constant challenge. Ingesting raw logs into SIEMs or analytics tools can be expensive, especially when a significant portion of that data consists of health checks, internal monitoring traffic, or redundant fields. By filtering this noise at the source, organizations can significantly reduce storage and ingestion costs. Furthermore, reshaping logs to match a unified schema simplifies correlation across different providers, making incident investigation faster and more reliable.

Security and compliance also benefit from pre-export redaction. Sending unmasked IP addresses or user agents to third-party storage can create privacy liabilities. Transformers allow teams to apply consistent redaction rules, such as masking the last two octets of an IP address, before the data leaves Cloudflare’s network. This ensures that sensitive information never reaches downstream systems, reducing the risk of accidental exposure and simplifying compliance audits.

What you can do

• Review your current Logpush jobs to identify fields that are consistently discarded or ignored by your analytics tools.
• Use Transformer Studio to draft SQL queries that filter out health check endpoints and internal hostnames.
• Implement regular expression-based redaction for any personally identifiable information or sensitive headers in your logs.
• Normalize field names across different CDN providers by adding static provider tags and renaming columns to match your internal schema.
• Monitor the Transformer Health view in the Logpush dashboard to track transformation duration and detect any failed jobs.
• Test your queries against sample records in the preview mode before attaching them to production Logpush jobs to ensure correct output.