python - Data locality via many queues in Celery? -


we're trying design distributed pipeline crunches large numbers of data chunks in parallel fashion. we're moving towards adopting celery, 1 of requirements need able map jobs nodes in cluster, e.g. if 1 node has access data chunk.

the first answer comes mind multiple queues, potentially 1 queue per node, large (~64) number of nodes. feasible, , efficient? celery queues lightweight? there better way?

the best answer i've found date here:

is celery appropriate use many small, distributed systems?

which suggests celery indeed fit use case. perhaps i'll update again when we've implemented.


Comments

Post a Comment

Popular posts from this blog

blackberry 10 - how to add multiple markers on the google map just by url? -

i have android application, want convert blackberry 10 , google map needed in app,but blackberry 10 not support googlemap , blackberry developer web site supply way replace it.the sample below: @override public void oncreate(bundle savedinstancestate) { super.oncreate(savedinstancestate); setcontentview(r.layout.main); webview webview = (webview) findviewbyid(r.id.mywebview); webview.getsettings().setjavascriptenabled(true); webview.loadurl("http://maps.google.com/?ll=36.97, -122&lci=bike&z=13&t=p"); } i cannot understand it,i not know how add multiple markers on googlemap via url? can me? this worked me map shown in webview.
Read more

php - guestbook returning database data to flash -

i've been struggling problem long , unable make work, hope can me it. first, know php files correct , mysql database works fine. in flash, able submit comment database comments not display. did many tests such making returned_database flash separately , worked fine - can see old comments. long combine submitting 1 , returning 1 together, flash doesn't show comments more. put lower version flash effects on frame thinking doesn't collaborate returning flash, submitting flash works fine. since pretty complex problem hard me copy actionscript code because on different frames in flash. if has ideas fixing maybe can send file you?
Read more

delphi - Dynamic file type icon -

i want register file type in windows , make program delphi draw icon of file according contents (something custom icon format). how can this? thanks nortd , user539484 comments, searched internet "shell icon handler delphi" , found sample here , , shell+ components has component this.
Read more