concurrency - Processing web pages concurrently with Ruby -

-

i trying process content of different pages given array of urls, using ruby thread. however, when trying open url error: #<socketerror: getaddrinfo: name or service not known>

this how trying it:

sites.each |site|     threads << thread.new(site) |url|         puts url         #web = open(url) { |i| i.read } # same issue opening web way         web = net::http.new(url, 443).get('/', nil)         lock.synchronize             new_md5[sites_hash[url]] = digest::md5.hexdigest(web)         end     end end

sites array of urls.

the same program sequential works alright:

sites.each { |site|     web = open(site) { |i| i.read }     new_md5 << digest::md5.hexdigest(web) }

what's problem?

ugh. you're going open thread every site have process? if have 10,000 sites?

instead, set limit on number of threads, , turn sites queue, , have each thread remove site, process , site. if there no more sites in queue, thread can exit.

the example in queue documentation started.

instead of using get , always retrieve entire body, use backing database keeps track of last time page processed. use head check see if page has been updated since then. if has, then get. reduce your, , their, bandwidth , cpu usage. it's being network citizen, , playing nice other people's toys. if don't play nice, might not let play them more.

i've written hundreds of spiders , site analyzers. i'd recommend should have backing database , use keep track of sites you're going read, when last read them, if or down last time tried page, , how many times you've tried reach them , down. (the last don't bang code's head on wall trying reach dead/down sites.)

i had 75 thread app read pages. each thread wrote findings database, and, if page needed processed, html written record in table. single app read table , did processing. easy single app stay ahead of 75 threads because they're dealing slow internet.

the big advantage using backend database, code can shut down, , it'll pick @ same spot, next site processed, if write correctly. can scale run on multiple hosts too.

regarding not being able find host:

some things see in code:

either of explain why using open works code doesn't. (i'm assuming you're using openuri in conjunction single-threaded code though don't show it, since open doesn't know url.)

in general, i'd recommend using typhoeus , hydra process large numbers of sites in parallel. typhoeus handle redirects also, along allowing use head requests. can set how many requests handled @ same time (concurrency) , automatically handles duplicate requests (memoization) redundant urls don't pounded.


Comments

Post a Comment

Popular posts from this blog

python - mat is not a numerical tuple : openCV error -

-
i have write down code showing error ma not getting it: please help: showing mat not numerical tuple: import cv import cv2 capture = cv2.videocapture("j.3gp") while(1): _, frame1 = capture.read() grayimage1 = cv2.cvtcolor(frame1, cv2.color_bgr2gray) _, frame2 = capture.read() grayimage2 = cv2.cvtcolor(frame2, cv2.color_bgr2gray) differenceimage = cv2.absdiff(grayimage1,grayimage2) thresholdimage = cv2.threshold(differenceimage,25,255,cv2.thresh_binary) cv2.imshow("difference image", differenceimage) cv2.imshow("threshold image", thresholdimage) cv2.imshow("image", frame1) k = cv2.waitkey(30) & 0xff error arising : ----------------------------------------------------------------------------------------- traceback (most recent call last): file "desk.py", line 15, in <module> cv2.imshow("threshold image", thresholdimage) typeerror: mat not numerical tuple ...
Read more

c# - MSAA finds controls UI Automation doesn't -

-
i'm working on automating windows application. i'm using teststack white framework. i've hit problem. program has 'window' object cannot see inside of. white shows no controls inside of it. inspect.exe shows no controls inside of either when running in ui automation mode. if switch inspect msaa see controls inside fine. there anyway me use msaa c# handle on these controls? if can identify msaa functions need, can use p/invoke call them c#. here's example article doing msaa: http://www.codeproject.com/articles/38906/ui-automation-using-microsoft-active-accessibility also, pinvoke.net can used identify iaccessible (msaa) functions: http://www.pinvoke.net/search.aspx?search=iaccessible&namespace=[all] here's answer along lines: msaa com-based? finally, alternative p/invoke, might able use tlbimp.exe create wrapper assembly oleacc.dll, , access msaa functions through it. i'm not sure if works msaa, it's worth try. as e...
Read more

wordpress - .htaccess: RewriteRule: bad flag delimiters -

-
pulling hair out on one. have following .htaccess file custom redirects in addition wordpress's default settings. when tested on local wamp server worked fine after moving production i'm not getting rewriterule: bad flag delimiters error in server log , after while site goes down internal server error any appreciated. in advance! <ifmodule mod_rewrite.c> rewriteengine on rewritebase / rewriterule ^rockford_weddings___welcome\.html$ /wedding/ [r=301,l] rewriterule ^blog/?$ /journal/ [r=301,l] rewriterule ^blog/(.*)$ /$1 [r=301,l] rewriterule ^portraitinvestment/?$ /portraitinvestment\.pdf [r=301,l] rewriterule ^weddinginvestment/?$ /weddinginvestment\.pdf [r=301,l] rewriterule ^holidaycards2014/?$ /holidaycards2014\.pdf [r=301,l] </ifmodule> # begin wordpress <ifmodule mod_rewrite.c> rewriteengine on rewritebase / rewriterule ^index\.php$ - [l] rewritecond %{request_filename} !-f rewritecond %{request_filename} !-d rewriterule . /index.php [l] </...
Read more