CentOS Question posted in
The official documentation can be found here.

Centos – Bootstrap Fails on Cassandra node – Unable to join the cluster

The bootstrap fails with following Stream Error.

Existing Cluster

6 Node – CentOS 7 – Cassandra version 4.0.11

The new node being added to the existing cluster – Ubuntu 22.04 LTS – Cassandra version 4.0.14

ERROR [Stream-Deserializer-/x.x.x.x:7000-7879f952] 2024-11-22 05:20:52,935 StreamSession.java:731 - [Stream #7d3711a0-a891-11ef-903f-c9da906269c9] Streaming error occurred on session with peer x.x.x.:7000
org.apache.cassandra.streaming.StreamReceiveException: java.lang.AssertionError: Invalid arguments: cell:43764 point:-2144718900 delta:1
    at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:60)
    at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:38)
    at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53)
    at org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:172)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.AssertionError: Invalid arguments: cell:43764 point:-2144718900 delta:1
    at org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder$Spool.tryCell(StreamingTombstoneHistogramBuilder.java:496)
    at org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder$Spool.tryAddOrAccumulate(StreamingTombstoneHistogramBuilder.java:471)
    at org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder.update(StreamingTombstoneHistogramBuilder.java:106)
    at org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder.update(StreamingTombstoneHistogramBuilder.java:93)
    at org.apache.cassandra.io.sstable.metadata.MetadataCollector.updateLocalDeletionTime(MetadataCollector.java:218)
    at org.apache.cassandra.io.sstable.metadata.MetadataCollector.update(MetadataCollector.java:184)
    at org.apache.cassandra.db.rows.Rows.collectStats(Rows.java:106)
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter$StatsCollector.applyToRow(BigTableWriter.java:284)
    at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:141)
    at org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:117)
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:216)
    at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
    at org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:107)
    at org.apache.cassandra.db.streaming.CassandraStreamReader.writePartition(CassandraStreamReader.java:206)
    at org.apache.cassandra.db.streaming.CassandraCompressedStreamReader.read(CassandraCompressedStreamReader.java:98)
    at org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
    at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:53)
    ... 5 common frames omitted

Tried to bootstrap multiple times, it fails with the same error

Tags:

2

Answers


  1. 0

    Your existing cluster is running Cassandra 4.0.11, while the new node is on Cassandra 4.0.14.

    While minor version differences are usually supported within the same major version (4.x), it’s possible that a bug fix or behavioral change introduced between these versions is causing this.

    In particular, the stack trace references:
    StreamingTombstoneHistogramBuilder This suggests an issue with tombstone metadata handling during SSTable streaming.

    This kind of error is uncommon for a minor version mismatch, but Cassandra has had bugs in earlier 4.x versions related to streaming and tombstone histograms, which were resolved in later updates.

    What I would suggest is to upgrade the existing cluster to 4.0.14, repair the data, and try again.

    An alternative is to downgrade the new version if upgrading the existing cluster is not an option.

    Login or Signup to reply.
  2. 0

    One thing your could try would be to set auto_bootstrap: false, and then join the node to the cluster. It will join empty, so you should immediately run a nodetool repair or rebuild on it. Perhaps running the joining and streaming processes separately will yield some results.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search