Multi-threaded parsing in python, how to implement?


Warning: count(): Parameter must be an array or an object that implements Countable in /home/styllloz/public_html/qa-theme/donut-theme/qa-donut-layer.php on line 274
0 like 0 dislike
9 views
Have 500 txt files, each file is 1000 links(separated by commas). All links belong to the same site.
How you can implement multi-threaded parsing on these pages at least 10-20 threads

I came up with this option, but don't know it true or not?

# Rough pseudo-code # Take out the first folder txt file # Retrieve from a file 10 links # For each link launches a separate thread # Store the result in database # Expected completion poslednogo flow # Sleep 10 seconds to a minute #Removed from the file, these 10 links # Get the next 10 links # and so on

This is a normal variant or is there a better and correct approach?
And what better to use multiprocessing or thread?
by | 9 views

2 Answers

0 like 0 dislike
scrapy
cat list*.txt | parallel --jobs 20 script.py
by
0 like 0 dislike
Better aiohttp
by

Related questions

0 like 0 dislike
3 answers
0 like 0 dislike
5 answers
0 like 0 dislike
1 answer
110,608 questions
257,186 answers
0 comments
35,405 users