Add WARN logs to make RRRE writer-thread deadlock visible - #251
Draft
tanyastickles wants to merge 76 commits into
Draft
Add WARN logs to make RRRE writer-thread deadlock visible#251tanyastickles wants to merge 76 commits into
tanyastickles wants to merge 76 commits into
Conversation
…sume shell implementation (not yet written upstream)
…tdCodec Co-authored-by: Charles Connell <cconnell@hubspot.com>
…rt discrete evaluations (will be in 2.7) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
…ta table isolation (will be in 2.7) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
…stem table isolation (will be in 2.7) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
…ll pointer (will be in 2.7) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
…ions more likely to be approved (will be in 2.7) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
…ent to the TableSkewCost (will be in 2.7) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
…es. Can squash this, or delete in 2.7 (#167) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
…ostFunction epsilon when evaluating whether a move improved costs (will be in 2.7) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
…ldn't use "UnknownException" (not yet written upstream) Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Ray Mattingly <rmattingly@apache.org> Co-authored-by: Hernan Gelaf-Romer <hgelafromer@hubspot.com>
…eckAndMutate methods (not yet written upstream) Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Nihal Jain <nihaljain@apache.org> Co-authored-by: Hernan Gelaf-Romer <hgelafromer@hubspot.com>
…n 2.7) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
…kup failures (will be in 2.6.4) Signed-off-by: Ray Mattingly <rmattingly@apache.org> Co-authored-by: Ray Mattingly <rmattingly@apache.org> Co-authored-by: Hernan Gelaf-Romer <hgelafromer@hubspot.com>
…ationEndpoint when only one replica found (will be in 2.6.4)
…ottlingExceptions … (will be in 2.6.4) * HBASE-29469 Add metrics with more detail for RpcThrottlingExceptions (apache#7214) Co-authored-by: skhillon <skhillon@hubspot.com> Signed-off by: cconnell <cconnell@apache.org> Reviewed by: kgeisz <kevin.j.geiszler@gmail.com> * Removing unnecessary sanitization * Remove unnecessary tests --------- Co-authored-by: skhillon <skhillon@hubspot.com>
…te information (will be in 2.6.4) Signed-off by: Ray Mattingly <rmattingly@apache.org>
…ng individual rows on demand (will be in 2.6.4) Signed-off by: Ray Mattingly <rmattingly@apache.org>
…t2 (not yet merged upstream) (#197)
…er when reloading configuration (will be in 2.6.4) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org> Signed-off-by: Charles Connell <cconnell@apache.org>
…irdparty is upgraded)
Co-authored-by: Hernan Gelaf-Romer <hgelafromer@hubspot.com>
…anager can lead to data loss (#227)
…etween backups (#224) Co-authored-by: Hernan Gelaf-Romer <hgelafromer@hubspot.com>
…#228) * Fix an issue with duplicated backups of inactive host WAL files * Skip meta region WAL files
… regionServer JMX Metric (apache#7481) (#229) * HBASE-29729 Add table descriptor hash
Co-authored-by: Hernan Gelaf-Romer <hgelafromer@hubspot.com>
…ta loss (#231) Co-authored-by: Hernan Gelaf-Romer <hgelafromer@hubspot.com>
… regions without moving (apache#7563) (apache#7611) (#232) (will be in 2.7) Signed-off-by: Ray Mattingly <rmattingly@apache.org> Co-authored-by: Alex Hughes <ajkh88@users.noreply.github.com> Co-authored-by: Alex Hughes <ahughes@hubspot.com>
Co-authored-by: Hernan Gelaf-Romer <hgelafromer@hubspot.com>
…verridden by replication peers (apache#7578) Co-authored-by: skhillon <skhillon@hubspot.com> Signed-off by: <taklwu@apache.org>
This commit condenses several redundant APIs used to access the backup history. No functional changes. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Co-authored-by: DieterDP <90392398+DieterDP-ng@users.noreply.github.com>
Restores the ordering of BackupAdmin#getHistory that was accidentally reversed in HBASE-29808. Extends & refactors TestBackupShowHistory to verify correct behavior. Fixes a possible FileNotFoundException in BackupUtils#getHistory. Merges BackupSystemTable#getBackupHistory with BackupSystemTable#getBackupInfos, to further simplify backup info retrieval. Optimized various usages of history retrieval. Clarified some javadoc regarding backup history retrieval. (cherry picked from commit ed04e2e) Co-authored-by: Dieter De Paepe <dieterdp@ngdata.com>
…n backup roots have different host coverage (#237) Co-authored-by: Hernan Gelaf-Romer <hgelafromer@hubspot.com>
…ression (#236) * Initial changes to allow partial cell read in WAL * Add tests * More tests * Add back comments * Remove redundant tests and simplify * Get tag value even if tag is out of bounds due to addition * Clean up a bit * Add explanatory comment * Initialize to empty * Resolve PR comment issue regarding LRU eviction for tags * Address PR review: two-phase rollback and remove unused dictType param Rollback in UndoableLRUDictionary previously restored nodes one at a time, doing remove/setContents/put on the content-based nodeToIndex HashMap. This could clobber entries when two nodes shared the same content during the restore (e.g., an evicted value re-added to a different slot). The fix restores all node state first, then rebuilds nodeToIndex from scratch. Also removes the unused dictType parameter from TagCompressionContext since every caller hardcodes LRUDictionary.class and we always need UndoableLRUDictionary for correctness. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: skhillon <skhillon@hubspot.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
… file switch detected in ReplicationSourceWALReader run loop (apache#7909) (#242) When ReplicationSourceWALReader.run() detects a WAL file switch via the switched() check, it enqueues an EOF batch but does not update currentPosition. If the outer loop restarts (e.g., due to WALEntryFilterRetryableException), the new WALEntryStream is created with the stale position from the old file, applied to the new file. This causes an infinite retry loop (EOFException: Cannot seek after EOF) and the corrupted position may be persisted to ZK, surviving restarts. The fix resets currentPosition to entryStream.getPosition() (which returns 0 after dequeueCurrentLog()) before enqueuing the EOF batch. Includes a regression test that reproduces the bug by using nb.capacity=1 to force EOF detection at line 153 (not inside readWALEntries), combined with a WALEntryFilterRetryableException on the first entry of the new file to trigger the outer loop restart. (cherry picked from commit e4f9c65) Signed-off-by: Duo Zhang <zhangduo@apache.org> Co-authored-by: skhillon <skhillon@hubspot.com>
…kups (#244) Co-authored-by: Hernan Gelaf-Romer <hgelafromer@hubspot.com>
…processors in region replication (#247) * Filter cell. extended attributes and split oversized batches in region replica replication Region replica replication was forwarding all extended attributes including large "cell."-prefixed ones added by CDC coprocessors that have no use for replicas. Strip these in replicate() before buffering to reduce memory and network overhead. Also add RPC size limit enforcement (matching HBaseInterClusterReplicationEndpoint) to split batches that exceed hbase.ipc.max.request.size, preventing permanently stalled replication. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix test compilation: add replicationRpcLimit parameter to test call sites Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add unit tests for splitBatches and extract as package-private static method Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Remove controller.reset() between batch RPCs reset() clears priority, callTimeout, tableName, and regionInfo which were set by the superclass before call() was entered. Since setCellScanner is called at the top of each loop iteration with the new batch's scanner, no cleanup is needed between iterations. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Revert batch-splitting changes, keep only cell. attribute filtering Scoping down to just the extended attribute filtering. The batch-splitting for oversized RPCs will be addressed separately. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: skhillon <skhillon@hubspot.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Hernan Gelaf-Romer <hgelafromer@hubspot.com>
When region-replica replication writer threads exhaust their RPC retry budget during an AZ partition, waitUntilDrained() enters a permanent silent deadlock — totalBuffered never reaches zero because dead threads cannot call doneWriting() for unprocessed region buffers. The shipper thread then blocks indefinitely with no log output, making the incident invisible until the backlog alert fires or an operator takes a thread dump. Two targeted log additions close the observability gap: - EntryBuffers.waitUntilDrained(): emit WARN every 30s (15 × 2000ms wait cycles) reporting elapsed time and bytes still buffered. Zero noise on the happy path; fires only when truly stuck. - OutputSink.restartWriterThreadsIfNeeded(): elevate the dead-thread replacement log from DEBUG to WARN so it is visible in production. With both logs in place, the next incident will produce either the restart-loop sequence (Path C) or the deadlock sequence (Path B) unambiguously, without requiring an RS restart to diagnose. See HBaseUsabilityPlanning#70 for full root-cause analysis. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
charlesconnell
force-pushed
the
hubspot-2.6
branch
5 times, most recently
from
August 9, 2026 22:57
791eac7 to
e414974
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
On the next `regionserver_region_replica_backlog_early` incident, there is currently no way to distinguish — from logs alone — whether the peer has entered:
Both paths result in zero WAL progress and a climbing backlog, but their recovery paths differ. Without this distinction, every incident ends in a blind RS restart.
Issue
BRAVE
Backwards Compatibility
Log-only change. No behavior change, no config changes, no new dependencies. Safe to roll back by reverting.
Rollout and Rollback Plan
Ships with the next RS rolling restart. No flag needed.
Automated Testing
No test changes required — these are observability-only additions with no branching logic.
Verification
On next `regionserver_region_replica_backlog_early` alert:
Expect Dependencies to Fail
None.
REVIEWERS: Please review both the code changes and the answers above, and validate that they match the expectations for BRAVE