python - pandas - Why can't DataFrame.apply be used to set some columns Categorical -


i have pandas dataframe, , i'd efficiently turn multiple columns categorical columns. first thought use pandas.dataframe.apply convert relevant columns. using following example data:

import pandas pd  pdf = pd.dataframe(dict(name=       ('earl', 'eve', 'alan', 'randall', 'danielle'),                         age=        (    29,    17,     73,        31,         62),                         gender=     (   'm',   'f',    'm',       'm',        'f'),                         nationality=(  'us',  'uk',  'can',     'can',       'us'),                         height=     ( 182.9, 167.6,  175.3,     170.2,      172.8)),                    columns=('name', 'age', 'gender', 'nationality', 'height')) pdf = pdf.set_index('name') >>> print(pdf)            age gender nationality  height name earl       29      m            182.9 eve        17      f          uk   167.6 alan       73      m         can   175.3 randall    31      m         can   170.2 danielle   62      f            172.8

you can see apply approach not working:

cat_list = {'gender', 'nationality'} set_cat_list = lambda x: x.astype('category') if x.name in cat_list else x dfa = pdf.apply(set_cat_list)  >>> print('applied subset: dtype={}'.format(dfa['gender'].dtype)) applied subset: dtype=object

this not throw error, silently converts column categorical @ point. , check it's firing correctly, added probe:

in_cl = lambda x: x.name in cat_list set_cat_list_alert = lambda x: (set_cat_list(x),                                 sys.stdout.write('{}: {}\n'.format(x.name, in_cl(x))))[0] dfa = pdf.apply(set_cat_list_alert) >>> print('applied subset: dtype={}'.format(dfa['gender'].dtype)) age: false age: false gender: true nationality: true height: false applied subset: dtype=object

evidently, fires off correctly, test see if approach can work @ all, tried converting all columns, , apparently works fine:

set_cat = lambda x: x.astype('category') dfb = pdf.apply(set_cat)  >>> print('applied whole frame: dtype={}'.format(dfb['gender'].dtype)) applied whole frame: dtype=category

finally, tried using for loop duplicate final result, make sure mixed categorical / non-categorical columns can coexist this:

dfc = pdf.copy() cat in cat_list:     dfc[cat] = pdf[cat].astype('category')  >>> print('for loop: dtype={}'.format(dfc['gender'].dtype)) loop: dtype=category

so question - why can't dataframe.apply() used set of these columns categorical? missing here?

this bug, indicated issue here, , fixed in upcoming 0.17.0 release, due first week of october.

you can install 0.17.0rc1 by:

conda install pandas -c pandas


Comments

Post a Comment

Popular posts from this blog

1111. appearing after print sequence - php -

morning, i have code takes string of shoe sizes, explodes , puts in table. code works , info displayed correctly on page after table there few 1's after table. i'm confused this. so table looks (in columns of 3 on webpage) available in following sizes: 3 4 5 6 9 10 111 here's codes, purpose of example $x = 3|4|6|10|9 function get_sizes_pipe($x) { $counter = 0; $splits = explode("|",$x); asort($splits); $x = print "<table class='greentable'>"; $x .= print "<thead><tr><th scope='col' colspan='3' abbr='starter'>available in following sizes:</th></tr></thead><tbody><tr>"; foreach ($splits $split) { $entry = print "<td>".$split."</td>"; if($counter == 2){ $entry .= print "</tr><tr>"; $counter =0; } else { $counter++; ...
Read more

java - WARN : org.springframework.web.servlet.PageNotFound - No mapping found for HTTP request with URI [/board/] in DispatcherServlet with name 'appServlet' -

this question has answer here: why spring mvc respond 404 , report “no mapping found http request uri […] in dispatcherservlet”? 3 answers i trying making simple bbs i'm encountering following error, warn : org.springframework.web.servlet.pagenotfound - no mapping found http request uri [/board/] in dispatcherservlet name 'appservlet' i have tried solve error many times, have failed. how can solve error? here web.xml, controllerand , servlet-context. my web.xml <?xml version="1.0" encoding="utf-8"?> <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"> <filter> <filter-nam...
Read more

Ruby on Rails, ActiveRecord, Postgres, UTF-8 and ASCII-8BIT encodings -

weirdest error, who’s challenge try , me? spent hours yesterday on , it’s magic. can't believe happening , it's driving me crazy. btw, using: rails 4.2.3 ruby 2.2.3 pg 0.18 encoding.default_internal = encoding.default_external = utf-8 config.encoding = 'utf-8' it has postgres, activerecord , encodings! turns out we’ve been getting these errors whenever create new users on database special characters: encoding::undefinedconversionerror: "\xc3" ascii-8bit utf-8 \xc3 may vary depending on characters. weird part we’ve set since beginning utf-8. makes no sense , digged it, , used script: user.all.each |user| user.attributes.each |name, value| if value.is_a? string puts user.email + name.encoding.to_s + value.encoding.to_s end end end and output of 1 of users, users same: dr_lottahelp@blahblah.comutf-8utf-8 dr_lottahelp@blahblah.comutf-8ascii-8bit dr_lottahelp@blahblah.comutf-8ascii-8bit dr_lottahelp@blahblah.comutf-8asci...
Read more