python - Django slowdown - thousands of unused datababase calls -


i have weird issue database consisting of 27,000 items in main table, , ~400,000 items in many-to-many linking items. database postgresql, running psycopg2, on django server on mod_wsgi on apache2.

this particular view slows down 10-20 seconds render.

i having hard time finding slowdown coming from. simple logging tells me view taking less 75 ms render, it's not view logic or templates. , sql queries being executed view take 35 ms total. there several seconds of operations can't see happening.

profiling tells me this: 

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)      5597    3.750    0.001    3.820    0.001 /usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py:52(execute) 303114/66508    2.518    0.000    5.290    0.000 /usr/lib/python2.7/copy.py:145(deepcopy)    297634    0.762    0.000    0.762    0.000 /usr/lib/python2.7/copy.py:267(_keep_alive)     16627    0.622    0.000    6.145    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py:242(clone) 44159/22079    0.449    0.000    2.531    0.000 /usr/lib/python2.7/copy.py:234(_deepcopy_tuple)      5597    0.423    0.000    4.653    0.001 /usr/local/lib/python2.7/dist-packages/django/db/backends/util.py:37(execute) 38706/33254    0.374    0.000    2.970    0.000 /usr/local/lib/python2.7/dist-packages/django/utils/tree.py:55(__deepcopy__)     10961    0.367    0.000    0.599    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/base.py:326(__init__)     16558    0.322    0.000    8.946    0.001 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:241(iterator)      5597    0.293    0.000    2.071    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py:56(as_sql)      5602    0.266    0.000    0.983    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py:1035(add_filter)      5597    0.232    0.000    0.540    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py:242(get_default_columns) 94039/83135    0.221    0.000    1.783    0.000 /usr/lib/python2.7/copy.py:226(_deepcopy_list)     44448    0.216    0.000    0.216    0.000 /usr/local/lib/python2.7/dist-packages/django/utils/datastructures.py:121(__init__)     22209    0.207    0.000    0.207    0.000 /usr/lib/python2.7/encodings/utf_8.py:15(decode)     22080    0.199    0.000    1.550    0.000 /usr/lib/python2.7/copy.py:306(_reconstruct)      5723    0.198    0.000   15.201    0.003 /usr/local/lib/python2.7/dist-packages/django/db/models/fields/related.py:367(__get__)      5597    0.194    0.000    0.323    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py:104(__init__)     16627    0.179    0.000    6.396    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:904(_clone)     66702    0.174    0.000    0.174    0.000 /usr/local/lib/python2.7/dist-packages/django/utils/tree.py:18(__init__)      5597    0.162    0.000    0.787    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py:162(get_columns)      5597    0.161    0.000    0.302    0.000 /usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py:160(_cursor)     44448    0.159    0.000    0.159    0.000 /usr/local/lib/python2.7/dist-packages/django/utils/datastructures.py:116(__new__)     11182    0.149    0.000    0.304    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py:854()     16627    0.145    0.000    0.315    0.000 /usr/local/lib/python2.7/dist-packages/django/utils/datastructures.py:136(__deepcopy__)     40271    0.140    0.000    0.209    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py:40(quote_name_unless_alias)     16558    0.131    0.000    7.713    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py:763(results_iter)     34669    0.129    0.000    0.129    0.000 /usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/operations.py:71(quote_name) 16796/11194    0.128    0.000    0.689    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/where.py:76(as_sql)     22224    0.123    0.000    0.446    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:36(__init__)         1    0.115    0.115    0.115    0.115 /usr/local/lib/python2.7/dist-packages/psycopg2/__init__.py:119(connect)      5437    0.112    0.000   14.141    0.003 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:374(get)      5598    0.111    0.000    1.232    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py:1235(add_q)         1    0.109    0.109    0.109    0.109 /usr/local/lib/python2.7/dist-packages/django/template/loaders/filesystem.py:33(load_template_source)      5602    0.108    0.000    0.561    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/where.py:148(make_atom)

so looks there thousands of calls being done base.py, , deepcopy, when @ connection.queries value view, there 7 total, , totalling 35 ms.

has run issue before? there no loops in view code, it's normal heuristics, pk-based lookup, , saving.


Comments

Post a Comment

Popular posts from this blog

python - How to create a legend for 3D bar in matplotlib? -

Image
given ax = plt.subplot() : ax.bar()[0] can passed plt.legend() . however, ax.bar3d() returns none . how create legend displayed bars? update: passing legend="stuff" ax.bar3d() , calling ax.legend() raises /usr/lib/python2.6/site-packages/matplotlib/axes.py:4368: userwarning: no labeled objects found. use label='...' kwarg on individual plots. warnings.warn("no labeled objects found. " you need use proxy artist legends not supported. this code: from mpl_toolkits.mplot3d import axes3d import matplotlib.pyplot plt import numpy np fig = plt.figure() ax = fig.add_subplot(111, projection='3d') x, y = np.random.rand(2, 100) * 4 hist, xedges, yedges = np.histogram2d(x, y, bins=4) elements = (len(xedges) - 1) * (len(yedges) - 1) xpos, ypos = np.meshgrid(xedges[:-1]+0.25, yedges[:-1]+0.25) xpos = xpos.flatten() ypos = ypos.flatten() zpos = np.zeros(elements) dx = 0.5 * np.ones_like(zpos) dy = dx.copy() dz = hist.flatten() ax...
Read more

java - Multi-Label Document Classification -

i have database in store data based upon following 3 fields: id, text, {labels}. note each text has been assigned more 1 label \ tag \ class. want build model (weka \ rapidminer \ mahout) able recommend \ classify bunch of labels \ tags \ classes given text. i have heard svm , naive bayes classifier, not sure whether support multi-label classification or not. guides me right direction more welcome! the basic multilabel classification method one-vs.-the-rest (ovr), called binary relevance (br). basic idea take off-the-shelf binary classifier, such naive bayes or svm, create k instances of solve k independent classification problems. in python-like pseudocode: for each class k: learner = svm(settings) # example labels = [class_of(x) == k x in samples] learner.learn(samples, labels) then @ prediction time, run each of binary classifiers on sample , collect labels predict positive. (both training , prediction can done in parallel, since problems assumed ...
Read more

php - Dynamic url re-writing using htaccess -

how can rewrite http://mydomain.com/test/master.php?catid=2&subcatid=8 http://mydomain.com/test/master.php/catname/subcat using .htaccess file. once rewritten, how can access these parameters in master.php? note: untested, let me know if doesn't work. rewriterule ^master.php/([a-za-z0-9_-]+)/([a-za-z0-9_-]+)$ /master.php?catid$1&subcatid=$2 [l]
Read more