search - Efficiently returning a field of all query hits in Lucene -


i have large lucene index, , queries can hit 5000 documents or so. storing application metadata in field in lucene (apart text contents), , need small metadata field 5000 hits. currently, code looks this:

mapfieldselector field = new mapfieldselector("metadata"); scoredoc[] hits = searcher.search(query, null, 10000).scoredocs; (int = 0; < hits.length; i++) {     int index_doc_id = hits[i].doc;     document hitdoc = searcher.doc(index_doc_id, field); // expensive esp disk-based lucene index     metadata = hitdoc.getfieldable("metadata").stringvalue(); }

however, terribly slow because each call searcher.doc() pretty expensive. there way "batch" fetch of field hits may more responsive? or other way make work faster? (the thing inside scoredoc appears lucene doc id, understand should not relied upon. otherwise have maintained lucene doc id -> metadata map on own.) thanks!

update: trying use fieldcache's this:

string metadatas[] = org.apache.lucene.search.fieldcache.default.getstrings(searcher.getindexreader(), "metadata");

when open index, , upon query:

int ldocid = hits[i].doc; string metadata = metadatas[ldocid];

this working me.

you're best bet on improving performance, reduce stored data as can. if have large content field stored in index, setting indexed only, rather stored improve performance. storing content external lucene, fetched after hit found in index, better idea.

there possiblity there exists better way end result looking for. i'm guessing 5000 sets of metadata aren't end result here. analysis may handled more on indexed data in lucene, instead of pulling out of index first. no idea, based on you've provided, whether possible in case, worth look.


Comments

Post a Comment

Popular posts from this blog

python - How to create a legend for 3D bar in matplotlib? -

Image
given ax = plt.subplot() : ax.bar()[0] can passed plt.legend() . however, ax.bar3d() returns none . how create legend displayed bars? update: passing legend="stuff" ax.bar3d() , calling ax.legend() raises /usr/lib/python2.6/site-packages/matplotlib/axes.py:4368: userwarning: no labeled objects found. use label='...' kwarg on individual plots. warnings.warn("no labeled objects found. " you need use proxy artist legends not supported. this code: from mpl_toolkits.mplot3d import axes3d import matplotlib.pyplot plt import numpy np fig = plt.figure() ax = fig.add_subplot(111, projection='3d') x, y = np.random.rand(2, 100) * 4 hist, xedges, yedges = np.histogram2d(x, y, bins=4) elements = (len(xedges) - 1) * (len(yedges) - 1) xpos, ypos = np.meshgrid(xedges[:-1]+0.25, yedges[:-1]+0.25) xpos = xpos.flatten() ypos = ypos.flatten() zpos = np.zeros(elements) dx = 0.5 * np.ones_like(zpos) dy = dx.copy() dz = hist.flatten() ax...
Read more

java - Multi-Label Document Classification -

i have database in store data based upon following 3 fields: id, text, {labels}. note each text has been assigned more 1 label \ tag \ class. want build model (weka \ rapidminer \ mahout) able recommend \ classify bunch of labels \ tags \ classes given text. i have heard svm , naive bayes classifier, not sure whether support multi-label classification or not. guides me right direction more welcome! the basic multilabel classification method one-vs.-the-rest (ovr), called binary relevance (br). basic idea take off-the-shelf binary classifier, such naive bayes or svm, create k instances of solve k independent classification problems. in python-like pseudocode: for each class k: learner = svm(settings) # example labels = [class_of(x) == k x in samples] learner.learn(samples, labels) then @ prediction time, run each of binary classifiers on sample , collect labels predict positive. (both training , prediction can done in parallel, since problems assumed ...
Read more

php - Dynamic url re-writing using htaccess -

how can rewrite http://mydomain.com/test/master.php?catid=2&subcatid=8 http://mydomain.com/test/master.php/catname/subcat using .htaccess file. once rewritten, how can access these parameters in master.php? note: untested, let me know if doesn't work. rewriterule ^master.php/([a-za-z0-9_-]+)/([a-za-z0-9_-]+)$ /master.php?catid$1&subcatid=$2 [l]
Read more