erlang - Mirrored queue performance factors -


we operate 2 dual-node brokers, each broker having quite different queues , workloads. each box has 24 cores (h/t) worth of xeon e5645 @ 2.4ghz 48gb ram, connected gigabit lan ~150μs latency, running rhel 5.6, rabbitmq 3.1, erlang r16b hipe off. we've tried hipe on made no noticeable performance impact, , crashy.

we appear have hit ceiling our message rates of between 1,000/s , 1,400/s both in , out. broker-wide, not per-queue. adding more consumers doesn't improve throughput overall, gives particular queue bigger slice of apparent "pool" of resource.

every queue mirrored across 2 nodes make broker. our publishers , consumers connect equally both nodes in persistant way. notice adsl-like asymmetry in rates too; if manage publish high rate of messages deliver rate drops high double digits. testing un-mirrored queue has higher throughput, expected. queues , exchanges durable, messages not persistent.

we'd know can improve situation. cpu on box fine, beam takes core , half 1 process, 80% each of 2 cores couple of processes. rest of box idle. using ~20gb of ram in userland system cache filling rest. io rates fine. network fine.

is there erlang/otp tuning can do? delegate_count default 16, explain in bit more detail please?

this difficult answer without knowing more how producers , consumers configured, client library you're using , on. discussed on irc (http://dev.rabbitmq.com/irclog/index.php?date=2013-05-22) minute ago, i'd suggest attempt reproduce topology using multicastmain java load test tool ships rabbitmq java client. can configure multiple producers/consumers, message sizes , on. can 5khz out of two-node cluster ha on desktop, may client (or application code) related issue.


Comments

Post a Comment

Popular posts from this blog

python - How to create a legend for 3D bar in matplotlib? -

Image
given ax = plt.subplot() : ax.bar()[0] can passed plt.legend() . however, ax.bar3d() returns none . how create legend displayed bars? update: passing legend="stuff" ax.bar3d() , calling ax.legend() raises /usr/lib/python2.6/site-packages/matplotlib/axes.py:4368: userwarning: no labeled objects found. use label='...' kwarg on individual plots. warnings.warn("no labeled objects found. " you need use proxy artist legends not supported. this code: from mpl_toolkits.mplot3d import axes3d import matplotlib.pyplot plt import numpy np fig = plt.figure() ax = fig.add_subplot(111, projection='3d') x, y = np.random.rand(2, 100) * 4 hist, xedges, yedges = np.histogram2d(x, y, bins=4) elements = (len(xedges) - 1) * (len(yedges) - 1) xpos, ypos = np.meshgrid(xedges[:-1]+0.25, yedges[:-1]+0.25) xpos = xpos.flatten() ypos = ypos.flatten() zpos = np.zeros(elements) dx = 0.5 * np.ones_like(zpos) dy = dx.copy() dz = hist.flatten() ax...
Read more

java - Multi-Label Document Classification -

i have database in store data based upon following 3 fields: id, text, {labels}. note each text has been assigned more 1 label \ tag \ class. want build model (weka \ rapidminer \ mahout) able recommend \ classify bunch of labels \ tags \ classes given text. i have heard svm , naive bayes classifier, not sure whether support multi-label classification or not. guides me right direction more welcome! the basic multilabel classification method one-vs.-the-rest (ovr), called binary relevance (br). basic idea take off-the-shelf binary classifier, such naive bayes or svm, create k instances of solve k independent classification problems. in python-like pseudocode: for each class k: learner = svm(settings) # example labels = [class_of(x) == k x in samples] learner.learn(samples, labels) then @ prediction time, run each of binary classifiers on sample , collect labels predict positive. (both training , prediction can done in parallel, since problems assumed ...
Read more

php - Dynamic url re-writing using htaccess -

how can rewrite http://mydomain.com/test/master.php?catid=2&subcatid=8 http://mydomain.com/test/master.php/catname/subcat using .htaccess file. once rewritten, how can access these parameters in master.php? note: untested, let me know if doesn't work. rewriterule ^master.php/([a-za-z0-9_-]+)/([a-za-z0-9_-]+)$ /master.php?catid$1&subcatid=$2 [l]
Read more