python - pandas read frame and subsequent type manipulation issue -
hello rookie pandas , have particular behavior want know reason for
sqlnew = 'select fund_no,start_date,end_date,cash,bond,small_cap,large_cap,international odsact.act_src_fund_mapping;' actualfundmapping = psql.read_frame(sqlnew,cnxn)
'everyworks fine until above'
actualfundmapping.dtypes:: fund_no object start_date object end_date object cash float64 bond float64 small_cap float64 large_cap float64 international float64
'so since want stuff datetime.datetime(2013,1,1) in actualfundmapping['start_date'] try changing dtype below
actualfundmapping['start_date'] = pd.to_datetime(str(actualfundmapping['start_date'])) actualfundmapping['end_date'] = pd.to_datetime(str(actualfundmapping['end_date'])) actualfundmapping['fund_no'] = actualfundmapping['fund_no'].astype(np.int64)
'but existence tests come false
datetime.datetime(2012,1,1) in actualfundmapping['start_date'] false
"that entry exists in sql table , have verified it
where as
str(datetime.datetime(2012,1,1)) in str(actualfundmapping['start_date']) true
'the same issue repeats following
np.int64(1000) in actualfundmapping['fund_no'] false
i guess not understanding representation pandas use internally. appreciated.
*******update*******
upon further advice made start_date column of sql table index column of dataframe.
actualfundmapping = psql.read_frame(sqlnew,cnxn,'start_date')
actualfundmapping.index
[2012-01-01 00:00:00, ..., 2012-01-01 00:00:00]
length: 895, freq: none, timezone: none
datetime.datetime(2012,1,1) in actualfundmapping false
datetime.datetime(2012,1,1) in actualfundmapping.index
false
np.datetime64(datetime.datetime(2012,1,1)) in actualfundmapping.index
false
by doing str(actualfundmapping['start_date'])
, converting entire column single string. believe pandas stores strings dtype object
(to avoid truncation issues arise when using numpy string dtypes). try using pd.to_datetime(actualfundmapping['start_date'])
. if raises error, there may mixed-type data in column, , have more detective work figure out individual values causing problems.
as unutbu suggests, need sure check membership in series values, not index.
Comments
Post a Comment