Skip to content

CloudNativePG query logs cannot be parsed if the query contains JSON #844

@jmealo

Description

@jmealo

Thank you for adding CNPG support here:
5aa95bc

This appears to be a compounding issue, Loki truncates log lines at 16KB.

With the auto_explain output, we exceed this by quite a bit.

I've observed that the following query cannot be parsed by PGBadger:

WITH hex_cells AS (
            SELECT h3_geography_to_cells_buffered(
                ST_GeomFromGeoJSON($1)::geography,
                $2,
                $3
            ) AS hex_id
        ),
        search_point AS (
            SELECT ST_GeomFromGeoJSON($4)::geography AS geog
        )
        SELECT DISTINCT ON (s.status_id)
               s.status_id,
               s.virtual,
               s.asset_id,
               s.asset_name,
               s.organization_id,
               ash.asset_status,
               ash.status_source AS asset_status_source,
               ash.created AS updated_asset_status,
               ash.expires_at AS asset_status_expires_at,
               s.asset_status_mtbf,
               s.asset_status_mttr,
               psh.power_status,
               psh.status_source AS power_status_source,
               psh.created AS updated_power_status,
               psh.expires_at AS power_status_expires_at,
               s.power_status_mtbf,
               s.power_status_mttr,
               s.created,
               s.deleted,
               ST_AsGeoJSON(l.location) AS geojson,
               l.thoroughfare,
               l.dependent_thoroughfare,
               l.premise,
               l.sub_premise,
               l.dependent_locality,
               l.locality,
               l.administrative_area,
               l.administrative_area_2,
               l.postal_code,
               l.country,
               ST_Distance(l.location, sp.geog) as distance
          FROM assets.statuses AS s
               JOIN assets.status_hexagons AS sh
                    ON s.status_id = sh.status_id
               JOIN locations.locations AS l
                    ON s.location_id = l.location_id
               CROSS JOIN search_point sp
               LEFT JOIN assets.asset_status_history AS ash
                    ON s.status_id = ash.status_id
                    AND s.asset_status_history_last_created = ash.created
               LEFT JOIN assets.power_status_history AS psh
                    ON s.status_id = psh.status_id
                    AND s.power_status_history_last_created = psh.created
               JOIN hex_cells hc
                    ON sh.hex_id = hc.hex_id
         WHERE ST_DWithin(
                   l.location,
                   ST_GeomFromGeoJSON($5)::geography,
                   $6
               )
           AND NOT s.replaced
        
                   AND s.organization_id = $7
            ORDER BY s.status_id, distance

It appears that queries that contain JSON (which at this point, is nested, due to the logging format) are not properly parsed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions