astyanax - Unexpected UnavailableException in a Cassandra cluster -

• i have 3-node c* cluster.
• c* client has read consistency level set quorum.
• when 1 node in cluster down getting unavailableexception in response read query

why? quorum cluster consisting of 3 nodes 2, should handle outage of 1 node.

a few more details:

cassandra version:

releaseversion: 1.1.6 

configuration of keyspace , column family:

keyspace: questionanswerservice:   replication strategy: org.apache.cassandra.locator.networktopologystrategy   durable writes: true     options: [datacenter1:2]   column families:   //...   columnfamily: answersbyquestion       key validation class: org.apache.cassandra.db.marshal.bytestype       default column value validator: org.apache.cassandra.db.marshal.bytestype       columns sorted by: org.apache.cassandra.db.marshal.bytestype       gc grace seconds: 864000       compaction min/max thresholds: 4/32       read repair chance: 1.0       dc local read repair chance: 0.0       populate io cache on flush: false       replicate on write: true       caching: keys_only       bloom filter fp chance: default       compaction strategy: org.apache.cassandra.db.compaction.sizetieredcompactionstrategy       compression options:         sstable_compression: org.apache.cassandra.io.compress.snappycompressor   //... 

exception thrown during read query when 1 node down:

2013-05-21 17:43:37 error countingconnectionpoolmonitor:81 - com.netflix.astyanax.connectionpool.exceptions.tokenrangeofflineexception: tokenrangeofflineexception: [host=cassandra.xxx.yyy(10.33.0.53):9160, latency=56(56), attempts=1]unavailableexception() com.netflix.astyanax.connectionpool.exceptions.tokenrangeofflineexception: tokenrangeofflineexception: [host=cassandra.xxx.yyy(10.33.0.53):9160, latency=56(56), attempts=1]unavailableexception()     @ com.netflix.astyanax.thrift.thriftconverter.toconnectionpoolexception(thriftconverter.java:165)     @ com.netflix.astyanax.thrift.abstractoperationimpl.execute(abstractoperationimpl.java:60)     @ com.netflix.astyanax.thrift.thriftcolumnfamilyqueryimpl$1$2.execute(thriftcolumnfamilyqueryimpl.java:198)     @ com.netflix.astyanax.thrift.thriftcolumnfamilyqueryimpl$1$2.execute(thriftcolumnfamilyqueryimpl.java:190)     @ com.netflix.astyanax.thrift.thriftsyncconnectionfactoryimpl$1.execute(thriftsyncconnectionfactoryimpl.java:136)     @ com.netflix.astyanax.connectionpool.impl.abstractexecutewithfailoverimpl.tryoperation(abstractexecutewithfailoverimpl.java:69)     @ com.netflix.astyanax.connectionpool.impl.abstracthostpartitionconnectionpool.executewithfailover(abstracthostpartitionconnectionpool.java:248)     @ com.netflix.astyanax.thrift.thriftcolumnfamilyqueryimpl$1.execute(thriftcolumnfamilyqueryimpl.java:188)     @ org.example.casstest$delayedinit$body.apply(casstest.scala:66)     @ scala.function0$class.apply$mcv$sp(function0.scala:34)     @ scala.runtime.abstractfunction0.apply$mcv$sp(abstractfunction0.scala:12)     @ scala.app$$anonfun$main$1.apply(app.scala:60)     @ scala.app$$anonfun$main$1.apply(app.scala:60)     @ scala.collection.linearseqoptimized$class.foreach(linearseqoptimized.scala:59)     @ scala.collection.immutable.list.foreach(list.scala:45)     @ scala.collection.generic.traversableforwarder$class.foreach(traversableforwarder.scala:30)     @ scala.app$class.main(app.scala:60)     @ org.example.casstest$.main(casstest.scala:14)     @ org.example.casstest.main(casstest.scala)     @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)     @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:39)     @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:25)     @ java.lang.reflect.method.invoke(method.java:597)     @ com.intellij.rt.execution.application.appmain.main(appmain.java:120) caused by: unavailableexception()     @ org.apache.cassandra.thrift.cassandra$get_slice_result.read(cassandra.java:7288)     @ org.apache.thrift.tserviceclient.receivebase(tserviceclient.java:78)     @ org.apache.cassandra.thrift.cassandra$client.recv_get_slice(cassandra.java:552)     @ org.apache.cassandra.thrift.cassandra$client.get_slice(cassandra.java:536)     @ com.netflix.astyanax.thrift.thriftcolumnfamilyqueryimpl$1$2.internalexecute(thriftcolumnfamilyqueryimpl.java:203)     @ com.netflix.astyanax.thrift.thriftcolumnfamilyqueryimpl$1$2.internalexecute(thriftcolumnfamilyqueryimpl.java:190)     @ com.netflix.astyanax.thrift.abstractoperationimpl.execute(abstractoperationimpl.java:55)     ... 22 more 

code in scala reproduces above error:

package org.example  import com.netflix.astyanax.connectionpool.impl.{countingconnectionpoolmonitor, connectionpoolconfigurationimpl} import com.netflix.astyanax.{keyspace, astyanaxcontext} import com.netflix.astyanax.impl.astyanaxconfigurationimpl import com.netflix.astyanax.connectionpool.nodediscoverytype import com.netflix.astyanax.retry.constantbackoff import com.netflix.astyanax.model.{columnfamily, consistencylevel} import com.netflix.astyanax.thrift.thriftfamilyfactory import com.netflix.astyanax.serializers.stringserializer import org.slf4j.loggerfactory import scala.collection.javaconversions._  object casstest extends app {    println("hello, cass-test")    val logger = loggerfactory.getlogger(casstest.getclass)    val clustername = "cassandra"   val hostname = "cassandra.xxx.yyy"   val port = 9160   val thriftsockettimeout = 4000   val keyspacename = "questionanswerservice"   val timeout = 5000    val connectionpool = new connectionpoolconfigurationimpl("connectionpool")     .setport(port)     //    .setmaxconnsperhost(1)     .setseeds(hostname + ":" + port)     .setsockettimeout(timeout)     .setconnecttimeout(timeout)     .settimeoutwindow(timeout)    val cassandracontext: astyanaxcontext[keyspace] =     new astyanaxcontext.builder()       .forcluster(clustername)       .withastyanaxconfiguration(new astyanaxconfigurationimpl()       .setdiscoverytype(nodediscoverytype.token_aware)       .setretrypolicy(new constantbackoff(timeout, 10000))       .setdefaultreadconsistencylevel(consistencylevel.cl_quorum))       .withconnectionpoolconfiguration(connectionpool)       .withconnectionpoolmonitor(new countingconnectionpoolmonitor())       .forkeyspace(keyspacename)       .buildkeyspace(thriftfamilyfactory.getinstance())    cassandracontext.start()    val keyspace: keyspace = cassandracontext.getentity()    val answersbyquestioncf = new columnfamily[string, string](     "answersbyquestion", // column family name     stringserializer.get(), // key serializer     stringserializer.get(), // column serializer     stringserializer.get()) // value serializer    while(true) {      logger.info("query start")      val result = keyspace       .preparequery(answersbyquestioncf)       .getkey("birthyear")       .execute()      logger.info("query finished: " + result.tostring)      result.getresult.getcolumnnames.take(10) foreach {       logger.info     }    }  } 

the number of nodes required consistency level function of replication factor, not number of nodes in cluster. rf=2, quorum 2 nodes must able read data.

with 3 nodes in cluster, rf=2 , reading @ cl.quorum, can access 1/3 of data 1 node down. reads other keys result in unavailable exception.

with 3 nodes in cluster, rf=3 , reading @ cl.quorum, can still access data 1 node down.