Skip to content

Conversation

sauravkumarrr
Copy link
Contributor

Fix memory leak in RedisKeysSourceTask

Problem

The sourceOffsets list in RedisKeysSourceTask was continuously growing without ever being cleared, causing a memory leak in long-running connectors.

Root Cause

  • commitRecord() method added offsets to the list via addSourceOffset()
  • commit() method was a no-op and never cleared the list
  • The collected offsets were never actually used for any functionality

Solution

  • Removed the unused sourceOffsets field and addSourceOffset() method
  • Simplified commitRecord() to be a no-op with explanatory comment
  • No functional impact since Redis key monitoring doesn't require offset acknowledgment

Why This Is Safe

Unlike RedisStreamSourceTask, the keys source connector uses Redis keyspace notifications which are fire-and-forget events that don't require acknowledgment. The RedisItemReader in LIVE mode handles state internally, making offset tracking unnecessary.

Fixes memory leak in production deployments monitoring high-volume Redis key changes.

@sauravkumarrr
Copy link
Contributor Author

Image

@sauravkumarrr
Copy link
Contributor Author

Heap dump analysis

Image Image

@jruaux
Copy link
Collaborator

jruaux commented Sep 3, 2025

Thanks for your contribution. I will merge this shortly

@jruaux jruaux merged commit ad8e07d into redis-field-engineering:master Sep 3, 2025
1 check passed
@jruaux
Copy link
Collaborator

jruaux commented Sep 3, 2025

🎉 This issue has been resolved in v1.1.0 (Release Notes)

@jruaux jruaux added the released Issue has been released label Sep 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
released Issue has been released
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants