Python HDF5 – concurrency, compression & I/O performance
depending the type of data you store, how you do it, and how you want to retrieve, HDF5 can offer vastly better performance. Storing in an
HDFStoreas a single array, float data, compressed (in other words, not storing it in a format that allows for querying), will be stored/read amazing fast. Even storing in the table format (which slows down the write performance), will offer quite good write performance. You can look at this for some detailed comparsions (which is whatHDFStoreuses under the hood). http://www.pytables.org/moin, here’s a nice picture: