Have 500 txt files, each file is 1000 links(separated by commas). All links belong to the same site.
How you can implement multi-threaded parsing on these pages at least 10-20 threads
I came up with this option, but don't know it true or not?
# Rough pseudo-code # Take out the first folder txt file # Retrieve from a file 10 links # For each link launches a separate thread # Store the result in database # Expected completion poslednogo flow # Sleep 10 seconds to a minute #Removed from the file, these 10 links # Get the next 10 links # and so on
This is a normal variant or is there a better and correct approach?
And what better to use multiprocessing or thread?