python - luigi target for non-existent table -

-

i'm trying set simple table existence test luigi task using luigi.hive.hivetabletarget

i create simple table in hive make sure there:

create table test_table (a int);

next set target luigi:

from luigi.hive import hivetabletarget target = hivetabletarget(table='test_table')  >>> target.exists() true

great, next try table know doesn't exist make sure returns false.

target = hivetabletarget(table='test_table_not_here')  >>> target.exists()

and raises exception:

traceback (most recent call last):   file "<stdin>", line 1, in <module>   file "/usr/lib/python2.6/site-packages/luigi/hive.py", line 344, in exists     return self.client.table_exists(self.table, self.database)   file "/usr/lib/python2.6/site-packages/luigi/hive.py", line 117, in table_exists     stdout = run_hive_cmd('use {0}; describe {1}'.format(database, table))   file "/usr/lib/python2.6/site-packages/luigi/hive.py", line 62, in run_hive_cmd     return run_hive(['-e', hivecmd], check_return_code)   file "/usr/lib/python2.6/site-packages/luigi/hive.py", line 56, in run_hive     stdout, stderr) luigi.hive.hivecommanderror: ('hive command: hive -e use default; describe test_table_not_here failed error code: 17', '', '\nlogging initialized using configuration in jar:file:/opt/cloudera/parcels/cdh-5.2.0-1.cdh5.2.0.p0.36/jars/hive-common-0.13.1- cdh5.2.0.jar!/hive-log4j.properties\nok\ntime taken: 0.822 seconds\nfailed:  semanticexception [error 10001]: table not found test_table_not_here\n')

edited formatting clarity

i don't understand last line of exception. of course table not found, whole point of existence check. expected behavior or have configuration issue need work out?

okay looks may have been bug in latest tagged release (1.0.19) fixed on master branch. code responsible line:

stdout = run_hive_cmd('use {0}; describe {1}'.format(database, table)) return not "does not exist" in stdout

which changed in master be:

stdout = run_hive_cmd('use {0}; show tables "{1}";'.format(database, table)) return stdout , table in stdout

the latter works fine whereas former throws hivecommanderror.

if want solution without having update master branch, create own target class minimal effort:

from luigi.hive import hivetabletarget, run_hive_cmd  class myhivetarget(hivetabletarget):     def exists(self):         stdout = run_hive_cmd('use {0}; show tables "{1}";'.format(self.database, self.table))         return self.table in stdout

this produce desired output.


Comments

Post a Comment

Popular posts from this blog

python - mat is not a numerical tuple : openCV error -

-
i have write down code showing error ma not getting it: please help: showing mat not numerical tuple: import cv import cv2 capture = cv2.videocapture("j.3gp") while(1): _, frame1 = capture.read() grayimage1 = cv2.cvtcolor(frame1, cv2.color_bgr2gray) _, frame2 = capture.read() grayimage2 = cv2.cvtcolor(frame2, cv2.color_bgr2gray) differenceimage = cv2.absdiff(grayimage1,grayimage2) thresholdimage = cv2.threshold(differenceimage,25,255,cv2.thresh_binary) cv2.imshow("difference image", differenceimage) cv2.imshow("threshold image", thresholdimage) cv2.imshow("image", frame1) k = cv2.waitkey(30) & 0xff error arising : ----------------------------------------------------------------------------------------- traceback (most recent call last): file "desk.py", line 15, in <module> cv2.imshow("threshold image", thresholdimage) typeerror: mat not numerical tuple ...
Read more

c# - MSAA finds controls UI Automation doesn't -

-
i'm working on automating windows application. i'm using teststack white framework. i've hit problem. program has 'window' object cannot see inside of. white shows no controls inside of it. inspect.exe shows no controls inside of either when running in ui automation mode. if switch inspect msaa see controls inside fine. there anyway me use msaa c# handle on these controls? if can identify msaa functions need, can use p/invoke call them c#. here's example article doing msaa: http://www.codeproject.com/articles/38906/ui-automation-using-microsoft-active-accessibility also, pinvoke.net can used identify iaccessible (msaa) functions: http://www.pinvoke.net/search.aspx?search=iaccessible&namespace=[all] here's answer along lines: msaa com-based? finally, alternative p/invoke, might able use tlbimp.exe create wrapper assembly oleacc.dll, , access msaa functions through it. i'm not sure if works msaa, it's worth try. as e...
Read more

wordpress - .htaccess: RewriteRule: bad flag delimiters -

-
pulling hair out on one. have following .htaccess file custom redirects in addition wordpress's default settings. when tested on local wamp server worked fine after moving production i'm not getting rewriterule: bad flag delimiters error in server log , after while site goes down internal server error any appreciated. in advance! <ifmodule mod_rewrite.c> rewriteengine on rewritebase / rewriterule ^rockford_weddings___welcome\.html$ /wedding/ [r=301,l] rewriterule ^blog/?$ /journal/ [r=301,l] rewriterule ^blog/(.*)$ /$1 [r=301,l] rewriterule ^portraitinvestment/?$ /portraitinvestment\.pdf [r=301,l] rewriterule ^weddinginvestment/?$ /weddinginvestment\.pdf [r=301,l] rewriterule ^holidaycards2014/?$ /holidaycards2014\.pdf [r=301,l] </ifmodule> # begin wordpress <ifmodule mod_rewrite.c> rewriteengine on rewritebase / rewriterule ^index\.php$ - [l] rewritecond %{request_filename} !-f rewritecond %{request_filename} !-d rewriterule . /index.php [l] </...
Read more