python - pandas read frame and subsequent type manipulation issue -
hello rookie pandas , have particular behavior want know reason for
sqlnew = 'select fund_no,start_date,end_date,cash,bond,small_cap,large_cap,international odsact.act_src_fund_mapping;' actualfundmapping = psql.read_frame(sqlnew,cnxn) 'everyworks fine until above'
actualfundmapping.dtypes:: fund_no object start_date object end_date object cash float64 bond float64 small_cap float64 large_cap float64 international float64 'so since want stuff datetime.datetime(2013,1,1) in actualfundmapping['start_date'] try changing dtype below
actualfundmapping['start_date'] = pd.to_datetime(str(actualfundmapping['start_date'])) actualfundmapping['end_date'] = pd.to_datetime(str(actualfundmapping['end_date'])) actualfundmapping['fund_no'] = actualfundmapping['fund_no'].astype(np.int64) 'but existence tests come false
datetime.datetime(2012,1,1) in actualfundmapping['start_date'] false "that entry exists in sql table , have verified it
where as
str(datetime.datetime(2012,1,1)) in str(actualfundmapping['start_date']) true 'the same issue repeats following
np.int64(1000) in actualfundmapping['fund_no'] false i guess not understanding representation pandas use internally. appreciated.
*******update*******
upon further advice made start_date column of sql table index column of dataframe.
actualfundmapping = psql.read_frame(sqlnew,cnxn,'start_date')
actualfundmapping.index
[2012-01-01 00:00:00, ..., 2012-01-01 00:00:00]
length: 895, freq: none, timezone: none
datetime.datetime(2012,1,1) in actualfundmapping false datetime.datetime(2012,1,1) in actualfundmapping.index
false
np.datetime64(datetime.datetime(2012,1,1)) in actualfundmapping.index
false
by doing str(actualfundmapping['start_date']), converting entire column single string. believe pandas stores strings dtype object (to avoid truncation issues arise when using numpy string dtypes). try using pd.to_datetime(actualfundmapping['start_date']). if raises error, there may mixed-type data in column, , have more detective work figure out individual values causing problems.
as unutbu suggests, need sure check membership in series values, not index.
Comments
Post a Comment