Schema migrator fails on external ClickHouse cluster

[Kinue72]

Bug description

When running SigNoz schema migration against an external ClickHouse cluster with 1 shard and 3 replicas on Talos Linux, the migration process fails.

How to reproduce

1. Create Clickhouse Cluster using Altinity Operator (1 Shard, 3 Replication)
2. Using helm chart to deploy signoz

Version information

• Signoz version: 0.92.1 (Helm Chart)
• Browser version: Brave
• Your OS and version: Talos Linux v1.11.1
• Your CPU Architecture(ARM/Intel): x64

Additional context

# kubectl logs -n monitoring signoz-schema-migrator-sync-init-lsjdr
Defaulted container "schema-migrator" out of: schema-migrator, schema-migrator-sync-init (init)
{"L":"info","timestamp":"2025-09-24T17:31:57.402Z","C":"signozschemamigrator/main.go:91","M":"Running migrations in sync mode","dsn":"tcp://signoz_monitoring:fancy_password@clickhouse-cluster.clickhouse-cluster:9000","replication":true,"cluster-name":"cluster"}
{"L":"info","timestamp":"2025-09-24T17:31:57.402Z","C":"signozschemamigrator/main.go:104","M":"Up migrations","versions":[]}
{"L":"info","timestamp":"2025-09-24T17:31:57.402Z","C":"signozschemamigrator/main.go:117","M":"Down migrations","versions":[]}
{"L":"info","timestamp":"2025-09-24T17:31:57.402Z","C":"signozschemamigrator/main.go:127","M":"Parsed DSN","optsError":"json: unsupported type: func(context.Context, string) (net.Conn, error)"}
{"L":"info","timestamp":"2025-09-24T17:31:57.402Z","C":"signozschemamigrator/main.go:133","M":"Opened connection"}
{"L":"info","timestamp":"2025-09-24T17:31:57.824Z","C":"schema_migrator/manager.go:157","M":"Created databases","dbs":["signoz_traces","signoz_metrics","signoz_logs","signoz_metadata","signoz_analytics","signoz_meter"]}
{"L":"info","timestamp":"2025-09-24T17:31:57.824Z","C":"schema_migrator/manager.go:166","M":"Creating schema migrations tables"}
{"L":"info","timestamp":"2025-09-24T17:31:57.824Z","C":"schema_migrator/manager.go:949","M":"Running operation","migration_id":1,"database":"signoz_logs","skip_status_update":true}
{"L":"info","timestamp":"2025-09-24T17:31:57.824Z","C":"schema_migrator/manager.go:959","M":"Waiting for running mutations before running the operation"}
{"L":"info","timestamp":"2025-09-24T17:31:57.826Z","C":"schema_migrator/manager.go:389","M":"Connecting to new host","host":"10.0.2.39:9440"}
{"L":"info","timestamp":"2025-09-24T17:31:57.826Z","C":"schema_migrator/manager.go:944","M":"Updating migration entry","query":"ALTER TABLE signoz_logs.schema_migrations_v2 ON CLUSTER cluster UPDATE status = $1, error = $2, updated_at = $3 WHERE migration_id = $4","status":"failed","error":"failed to get conn\n[handshake] unexpected packet [21] from server","migration_id":1}
Error: failed to bootstrap migrations: failed to create schema_migrations_v2 table
failed to get conn
[handshake] unexpected packet [21] from server
code: 60, message: There was an error on [chi-cluster-0-2:9440]: Code: 60. DB::Exception: Could not find table: schema_migrations_v2. (UNKNOWN_TABLE) (version 25.8.6.11 (official build))
Usage:
  signoz-schema-migrator sync [flags]

Flags:
      --down string   Down migrations to run, comma separated. Must provide down migrations explicitly to run
  -h, --help          help for sync
      --up string     Up migrations to run, comma separated. Leave empty to run all up migrations

Global Flags:
      --cluster-name string   Cluster name to use while running migrations (default "cluster")
      --dev                   Development mode
      --dsn string            Clickhouse DSN
      --replication           Enable replication

Thank you for your bug report – we love squashing them!

[allan-opsguru]

I got the same errors. But figured out that it was caused by DB corruption from my previous installations.

Try removing all PVCs in Signoz namespace and redeploy it again.

[Kinue72]

I got the same errors. But figured out that it was caused by DB corruption from my previous installations.

Try removing all PVCs in Signoz namespace and redeploy it again.

After digging into the source code, I found that the migrator uses the database port with SSL, while the initial connection uses non-SSL. I’ve already submitted a PR to the Helm chart to support SSL without requiring a valid certificate.