python - Analyzing huge dataset from 80GB file with only 32GB of RAM -


i have huge text file ( around 80g ). file contains numbers(integers+floats) , has 20 columns. have analyze each column. analyze mean, have basic calculations on each column finding mean, plotting histograms, check if condition satisfied or not etc. reading file following 

with open(filename) original_file:         all_rows = [[float(digit) digit in line.split()] line in original_file]     all_rows = np.asarray(all_rows)

after analysis on specific columns. use 'good' configuration server/workstation (with 32gb ram) execute program. problem not able finish job. waited day finish program still running after 1 day. had kill manually later on. know script correct without error because have tried same script on smaller size files (around 1g) , worked nicely.

my initial guess have memory problem. there way can run such job? different method or other way ?

i tried splitting files smaller file size , analyzed them individually in loop follows

pre_name = "split_file"    k in range(11):  #there 10 files 8g each         filename = pre_name+str(k).zfill(3) #my files in form "split_file000, split_file001 ..."         open(filename) original_file:             all_rows = [[float(digit) digit in line.split()] line in original_file]         all_rows = np.asarray(all_rows)         #some analysis here         plt.hist(all_rows[:,8],100)  #plotting histogram 9th column all_rows = none

i have tested above code on bunch of smaller files , works fine. again same problem when used on big files. suggestions? there other cleaner way ?

for such lengthy operations (when data don't fit in memory), might useful use libraries dask ( http://dask.pydata.org/en/latest/ ), particularly dask.dataframe.read_csv read data , perform operations in pandas library (another useful package mention).


Comments

Post a Comment

Popular posts from this blog

html - Outlook 2010 Anchor (url/address/link) -

i'm trying link specific pdf page msoutlook'10 or msword'10, #page=... anchor particularly useful , works fine elsewhere. unfortunately when click [ctr-k] in outlook or winword, links re-formatted assuming unc path. pdf file located on shared drive. using "file:" prefix links open pdf in adobe reader (outside of browser), , "http:" open inside browser. issue seem need open in browser "file:" prefix in order anchor work. know anchors work inside browser. outlook uses windows/explorer default program (based on file extension) seems open in adobe reader neglecting anchor. is aware of solution, link specific page or bookmark of shared-pdf ms outlook and/or winword link? common references: mailermailer.com/...anchor-tags-html-emails campaignmonitor.com/...anchor-links-in-email-newsletters/ stackoverflow.com/...anchor-in-pdf-document
Read more

javascript - Why does running this loop 9 times take 100x longer than running it 8 times? -

consider code: test = function() { } t = new test(); (var = 0; < 8; i++) { result = t instanceof test; } if change number of iterations 8 9 , loop take 100 times longer complete in latest version of firefox (41.0.1). tested on 2 different pcs , magic limit 8. here jsperf test used: http://jsperf.com/instanceof-8-times-vs-9-times does have idea why might happen? seems specific instanceof . not happen if else object, example check property. note: filed bugzilla bug this. jan de mooij mozilla team has posted details in bugzilla thread . here's simplistic interpretation of highly technical answers: in i < 8 case, firefox smart enough hoist result = t instanceof test; statement out of loop (according tests, doesn't seem omit altogether). in i < 9 case, apparently doesn't optimization . why? reason not clear, related fact 9 iterations threshold above function considered "hot" enough run through jit compiler. i < 8 cas...
Read more

Getting gateway time-out Rails app with Nginx + Puma running on Digital Ocean -

following this guide setup app on digital ocean droplet, not working. site not loading , it's throwing: "gateway time-out". the puma server works, since ps aux | grep puma prints: root 15878 0.0 0.4 376832 2168 ? sl 12:50 0:00 puma 2.13.4 (unix:///var/www/myapp/shared/tmp/sockets/myapp-puma.sock) [20150930164940] root 15881 0.0 0.7 903676 3692 ? sl 12:50 0:06 puma: cluster worker 0: 15878 [20150930164940] root 24654 0.0 0.1 11748 916 pts/2 s+ 18:00 0:00 grep --color=auto puma my nginx config: user root; worker_processes 1; events { worker_connections 1024; } http { upstream puma { server unix:///var/www/myapp/shared/tmp/sockets/myapp-puma.sock; } server { listen 80; server_name myapp.com; root /var/www/myapp/current/public; access_log /var/www/myapp/current/log/access.log; error_log /...
Read more