Removing Elements From 300MG Xml In Python / Element Tree -


i'm trying parse 300mb xml in elementtree, based on advise can python xml elementtree parse large xml file? 

from xml.etree import elementtree et  event, elem in et.iterparse('c:\...path...\desc2015.xml'):       if elem.tag == 'descriptorrecord':         e in elem._children:             if str(e.tag) in ['datecreated', 'year', 'month', 'treenumber', 'historynote', 'previousindexing']:                 e.clear()                 elem.remove(e)                 print 'removed %s' % e

giving...

removed <element 'historynote' @ 0x557cc7f0> removed <element 'datecreated' @ 0x557fa990> removed <element 'historynote' @ 0x55809af0> removed <element 'datecreated' @ 0x5580f5d0>

however, keeps going, file isn't getting smaller, , on inspection elements still there. tried either e.clear() or elem.remove(e), same results. regards

update

error code first comment on @alexanderlukanin13 s answer:

traceback (most recent call last): file "c:\users\eddie\downloads\eclipse\plugins\org.python.pydev_4.0.0.201504132356\pysrc\pydevd.py", line 1570, in trace_dispatch traceback (most recent call last): file "c:\users\eddie\downloads\eclipse\plugins\org.python.pydev_4.0.0.201504132356\pysrc\pydevd.py", line 2278, in globals = debugger.run(setup['file'], none, none) file "c:\users\eddie\downloads\eclipse\plugins\org.python.pydev_4.0.0.201504132356\pysrc\pydevd.py", line 1704, in run pydev_imports.execfile(file, globals, locals) # execute script file "c:\users\eddie\downloads\eclipse\plugins\org.python.pydev_4.0.0.201504132356\pysrc\runfiles.py", line 234, in main() file "c:\users\eddie\downloads\eclipse\plugins\org.python.pydev_4.0.0.201504132356\pysrc\runfiles.py", line 78, in main return pydev_runfiles.main(configuration) # note: still doesn't return proper value. file "c:\users\eddie\downloads\eclipse\plugins\org.python.pydev_4.0.0.201504132356\pysrc\pydev_runfiles.py", line 835, in main pydevtestrunner(configuration).run_tests() file "c:\users\eddie\downloads\eclipse\plugins\org.python.pydev_4.0.0.201504132356\pysrc\pydev_runfiles.py", line 762, in run_tests file_and_modules_and_module_name = self.find_modules_from_files(files) file "c:\users\eddie\downloads\eclipse\plugins\org.python.pydev_4.0.0.201504132356\pysrc\pydev_runfiles.py", line 517, in find_modules_from_files mod = self.__get_module_from_str(import_str, print_exception, pyfile) file "c:\users\eddie\downloads\eclipse\plugins\org.python.pydev_4.0.0.201504132356\pysrc\pydev_runfiles.py", line 476, in __get_module_from_str buf_err = pydevd_io.startredirect(keep_original_redirection=true, std='stderr') file "c:\users\eddie\downloads\eclipse\plugins\org.python.pydev_4.0.0.201504132356\pysrc\pydevd_io.py", line 72, in startredirect import sys memoryerror

the main problem in script don't save altered xml disk. need store reference root element , call elementtree.write:

from xml.etree import elementtree et  context = et.iterparse('input.xml') root = none event, elem in context:     if elem.tag == 'descriptorrecord':         e in list(elem.getchildren()):  # don't use _children, it's private field             if e.tag in ['datecreated', 'year', 'month', 'treenumber', 'historynote', 'previousindexing']:                 elem.remove(e)  # need remove(), not clear()     root = elem  open('output.xml', 'wb') file:     et.elementtree(root).write(file, encoding='utf-8', xml_declaration=true)

note: here use awkward (and unsafe) way root element - assume it's last element in iterparse output. if knows better way, please tell.


Comments

Post a Comment

Popular posts from this blog

html - Outlook 2010 Anchor (url/address/link) -

i'm trying link specific pdf page msoutlook'10 or msword'10, #page=... anchor particularly useful , works fine elsewhere. unfortunately when click [ctr-k] in outlook or winword, links re-formatted assuming unc path. pdf file located on shared drive. using "file:" prefix links open pdf in adobe reader (outside of browser), , "http:" open inside browser. issue seem need open in browser "file:" prefix in order anchor work. know anchors work inside browser. outlook uses windows/explorer default program (based on file extension) seems open in adobe reader neglecting anchor. is aware of solution, link specific page or bookmark of shared-pdf ms outlook and/or winword link? common references: mailermailer.com/...anchor-tags-html-emails campaignmonitor.com/...anchor-links-in-email-newsletters/ stackoverflow.com/...anchor-in-pdf-document
Read more

javascript - Why does running this loop 9 times take 100x longer than running it 8 times? -

consider code: test = function() { } t = new test(); (var = 0; < 8; i++) { result = t instanceof test; } if change number of iterations 8 9 , loop take 100 times longer complete in latest version of firefox (41.0.1). tested on 2 different pcs , magic limit 8. here jsperf test used: http://jsperf.com/instanceof-8-times-vs-9-times does have idea why might happen? seems specific instanceof . not happen if else object, example check property. note: filed bugzilla bug this. jan de mooij mozilla team has posted details in bugzilla thread . here's simplistic interpretation of highly technical answers: in i < 8 case, firefox smart enough hoist result = t instanceof test; statement out of loop (according tests, doesn't seem omit altogether). in i < 9 case, apparently doesn't optimization . why? reason not clear, related fact 9 iterations threshold above function considered "hot" enough run through jit compiler. i < 8 cas...
Read more

Getting gateway time-out Rails app with Nginx + Puma running on Digital Ocean -

following this guide setup app on digital ocean droplet, not working. site not loading , it's throwing: "gateway time-out". the puma server works, since ps aux | grep puma prints: root 15878 0.0 0.4 376832 2168 ? sl 12:50 0:00 puma 2.13.4 (unix:///var/www/myapp/shared/tmp/sockets/myapp-puma.sock) [20150930164940] root 15881 0.0 0.7 903676 3692 ? sl 12:50 0:06 puma: cluster worker 0: 15878 [20150930164940] root 24654 0.0 0.1 11748 916 pts/2 s+ 18:00 0:00 grep --color=auto puma my nginx config: user root; worker_processes 1; events { worker_connections 1024; } http { upstream puma { server unix:///var/www/myapp/shared/tmp/sockets/myapp-puma.sock; } server { listen 80; server_name myapp.com; root /var/www/myapp/current/public; access_log /var/www/myapp/current/log/access.log; error_log /...
Read more