I’m trying to drag my organization out of the dark ages so we can support IPv6. Cassandra is giving me fits. In cassandra.yaml, I have set listen_address to "[fc00:…]" (with the quotes and square brackets) (yes, magical non-public addresses in our test network), set rpc_address to "[0::0]", set all of the prefer_ipv4 to false (irrelevant with listen_address?), and elsewhere removed all of the "PreferIPv4Stack=true" command line settings.
None of the "escape : with backslash" things I tried passed the config parser. This version does. However, a bit later, the PropertyFileSnitch constructor calls reloadConfiguration which calls InetAddress.getByName("fc00") (I believe) and that causes an Exception Caused by: java.net.UnknownHostException: fc00: Name or service not known reported in cassandra.log. My presumption is that some code in the stack already stripped off the square brackets, and intended to strip off a port number, but wound up stripping off most of the IPv6 address; I’ve had to deal with that elsewhere. The doc on getByName indicates that it should be able to handle this address format (with or without the square brackets), but it appears to be treating it as a hostname (reasonable if the address has been stripped like I think it has).
So, does PropertyFileSnitch get that IP address ultimately from listen_address? In other words, do I have another config file that I haven’t found and escaped properly?
Am I really getting the IPv6 format right for listen_address?
What haven’t I noticed?
This is cassandra-3.11.6-1 on CentOS 7; from answers elsewhere, that’s likely relevant.
NOTE: I’m the platform guy, I do NOT know Cassandra, so simple answers would be welcomed, including full pathnames of any config files.
Thanks!
UPDATE: it looks like just plain "fd00:…" works for listen_address (yes, we changed it), but now I’m getting Address Family unavailable in MessagingService.listen(). There are other things with IPv6 listening sockets, several ESTAB TCP6 connections, the sysctls are all set right, so once again I’m kinda stumped.
By the way, what sort of escaping do IPv6 addresses in the topology files need? There’s a comment about it, but no specifics.
UPDATE: It appears that the "Protocol family unavailable" error is from the address in the topology property file.
2
Answers
Found the problem! We had an /etc/cassandra/default.conf/kvm.options that included -Djava.net.preferIPv4Stack=true that I hadn't found, because it's from the cassandra package, instead of our cassandra management package like the other config files I was working with. I added a JVM_OPTS="$JVM_OPTS -D...=false" in our cassandra-env.sh, and everything is so much better now!
The only place where an IPv6 address seemed to need escaping was in cassandra-topology.properties, where it seems that
(fully escaped and wrapped) is what works.
I'm STILL not a cassandra or even a java person. I got handed this because I've done this sort of thing for several other parts of our software suite.
I have a feeling the problem is that you’re not using real IP addresses and it fails when Cassandra attempts to do a host lookup and therefore leading to the
UnknownHostException.FWIW the
listen_addressis the private network nodes use for internode communications (internal node-to-node messaging) so they don’t need to be public IPs. And this format is fine:If you post the full error message + full stack trace, I’d be happy to have a look. The Cassandra version would also be good to know. Cheers!