python - What is the advantage of PyTables? -

i have started learning pytables , found interesting. question is:

• what basic advantages of pytables on database(s) when comes huge datasets?
• what basic purpose of package (i can same sort of structuring in numpy , pandas, what's big deal pytables)?
• is helpful in analysis of big datasets? can elaborate of example , comparisons?

thank all.

what basic advantages of pytables on database(s) when comes huge datasets?

effectively, is database. of course it's hierarchical database rather 1-level key-value database dbm (which less flexible) or relational database sqlite3 (which more powerful, more complicated).

but main advantage on non-numerics-specific database same advantage of, say, numpy ndarray on plain python list. it's optimized performing lots of vectorized numeric operations, if that's you're doing it, it's going take less time , space.

what basic purpose of package

quoting first line of the front page (or, if prefer, first line of the faq):

pytables package managing hierarchical datasets , designed efficiently , cope extremely large amounts of data.

there's page listing mainfeatures, linked near top of front page.

(i can same sort of structuring in numpy , pandas, what's big deal pytables)?

really? can handle 64gb of data in numpy or pandas on machine 16gb of ram? or 32-bit machine?

no, can't. unless split data bunch of separate sets load, process, , save needed—but that's going more complicated, , slower.

it's asking why need numpy when can same thing regular python list , iterators. pure python great when have array of 8 floats, not when have 10000x10000 array of them. , numpy great when have couple of 10000x10000 arrays, not when have dozen interconnected arrays ranging 20gb in size.

is helpful in analysis of big datasets?

yes.

can elaborate of example…

yes. rather copying of examples here, why don't @ simple examples on front page of docs, slew of examples in source tree, links real-world use cases 2 clicks front page of docs, etc.?

if want convince of usefulness of pytables, take of examples , scale 32gb worth of data, try figure out how you'd exact same thing in numpy or pandas.