mysql - Load data infile slow executing from node js -


i'm trying insert more 600,000 rows in table script written in node js using next query. 

var sql =   " load data local infile '/tmp/insertfile18.csv'" +                      " table `pricing_leasing`"  +                       " fields terminated ','  lines terminated '+-'";

the problem never finishes (i waited 1 hour on lunch), neither give error. executed directly in bash , fast, less 1 minute. decided write little script in nodejs doesn't else except execute script , finishes in 2 minutes ok.

my question why if execute query in bash or in it's own node js script work fine, if have more stuff going on before (other queries on different tables & file operations) became slow.

update

function uploadfile() {  var deferred    = global.q.defer(),     strfilename = "/tmp/insertfile" + global.company.id + ".csv",     cb = function ( err ) {         //fs.unlinksync( strfilename );         if ( err ) {             console.log( err );              global.objrstofinsert['success'] = false;         } else {             // update return structure success message             global.objrstofinsert['success'] = true;         }         console.log("finished uploads");         deferred.resolve();     };      var lstinsertpricingcolumns = '(`sourcecompany_id`,'+                             '`company_id`,'+                             '`price`,`active`,`row_hash`)';      var sql =   " load data local infile '" + strfilename + "'" +                  " table " + global.company.tableinsert +                    " fields terminated ','  lines terminated '+-'"+                  lstinsertpricingcolumns;      global.db['rates'].query( sql, cb );     return deferred.promise; }

the problem cloud server have less ram local. guess file big. fixed splitting in multiples smaller files load data , work fine.

thanks.


Comments

Post a Comment

Popular posts from this blog

1111. appearing after print sequence - php -

morning, i have code takes string of shoe sizes, explodes , puts in table. code works , info displayed correctly on page after table there few 1's after table. i'm confused this. so table looks (in columns of 3 on webpage) available in following sizes: 3 4 5 6 9 10 111 here's codes, purpose of example $x = 3|4|6|10|9 function get_sizes_pipe($x) { $counter = 0; $splits = explode("|",$x); asort($splits); $x = print "<table class='greentable'>"; $x .= print "<thead><tr><th scope='col' colspan='3' abbr='starter'>available in following sizes:</th></tr></thead><tbody><tr>"; foreach ($splits $split) { $entry = print "<td>".$split."</td>"; if($counter == 2){ $entry .= print "</tr><tr>"; $counter =0; } else { $counter++; ...
Read more

java - WARN : org.springframework.web.servlet.PageNotFound - No mapping found for HTTP request with URI [/board/] in DispatcherServlet with name 'appServlet' -

this question has answer here: why spring mvc respond 404 , report “no mapping found http request uri […] in dispatcherservlet”? 3 answers i trying making simple bbs i'm encountering following error, warn : org.springframework.web.servlet.pagenotfound - no mapping found http request uri [/board/] in dispatcherservlet name 'appservlet' i have tried solve error many times, have failed. how can solve error? here web.xml, controllerand , servlet-context. my web.xml <?xml version="1.0" encoding="utf-8"?> <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"> <filter> <filter-nam...
Read more

Ruby on Rails, ActiveRecord, Postgres, UTF-8 and ASCII-8BIT encodings -

weirdest error, who’s challenge try , me? spent hours yesterday on , it’s magic. can't believe happening , it's driving me crazy. btw, using: rails 4.2.3 ruby 2.2.3 pg 0.18 encoding.default_internal = encoding.default_external = utf-8 config.encoding = 'utf-8' it has postgres, activerecord , encodings! turns out we’ve been getting these errors whenever create new users on database special characters: encoding::undefinedconversionerror: "\xc3" ascii-8bit utf-8 \xc3 may vary depending on characters. weird part we’ve set since beginning utf-8. makes no sense , digged it, , used script: user.all.each |user| user.attributes.each |name, value| if value.is_a? string puts user.email + name.encoding.to_s + value.encoding.to_s end end end and output of 1 of users, users same: dr_lottahelp@blahblah.comutf-8utf-8 dr_lottahelp@blahblah.comutf-8ascii-8bit dr_lottahelp@blahblah.comutf-8ascii-8bit dr_lottahelp@blahblah.comutf-8asci...
Read more