How to be registred on the website Python 3 Requests?


Warning: count(): Parameter must be an array or an object that implements Countable in /home/styllloz/public_html/qa-theme/donut-theme/qa-donut-layer.php on line 274
0 like 0 dislike
6 views
Good evening. There is, for example, a website with a login form:
https://lolzteam.net/login

I don't quite understand how to be an authorized user with web scraping this resource. For example, I want to make a retrieve from multiple pages, but there is, they say "You can't see the text, maybe not logged in". I'd like to understand how to get around it. If possible, example, please.

import requests from bs4 import BeautifulSoup, NavigableString, Tag header = { 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Encoding': 'gzip, deflate', 'Accept-Language': 'EN-us,EN;q=0.8,en-US;q=0.5,en;q=0.3', 'Connection': 'keep-alive', 'X-Requested-With': 'XMLHttpRequest' } payload = { 'login': 'Some name', 'password': 'Some pass' } blacklist_lolz = 'https://lolzteam.net/forums/774/' main_lolz = 'https://lolzteam.net/' def beautifulsoup(url_content): return BeautifulSoup(url_content, 'lxml') with requests.Session() as s: p = s.post('https://lolzteam.net/login', data=payload) print(p.text) req = s.get(blacklist_lolz) print(req.text) soup = beautifulsoup(req.content) divs_all = soup.find_all('a', class_='PreviewTooltip ') links_all = [div['href'] for div in divs_all] for link in links_all: number_list = [] link_page = requests.get(main_lolz + link, headers=header) link_soup = beautifulsoup(link_page.content) # Some code ...


Why do I feel like I'm not doing right. In a loop, I go to each link and watch them sometimes, "You are not logged in, can't read text, and so on and so on".
by | 6 views

1 Answer

0 like 0 dislike
Last cycle you out of the session. In theory it should help
import requests from bs4 import BeautifulSoup, NavigableString, Tag header = { 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Encoding': 'gzip, deflate', 'Accept-Language': 'EN-us,EN;q=0.8,en-US;q=0.5,en;q=0.3', 'Connection': 'keep-alive', 'X-Requested-With': 'XMLHttpRequest' } payload = { 'login': 'Some name', 'password': 'Some pass' } blacklist_lolz = 'https://lolzteam.net/forums/774/' main_lolz = 'https://lolzteam.net/' def beautifulsoup(url_content): return BeautifulSoup(url_content, 'lxml') with requests.Session() as s: p = s.post('https://lolzteam.net/login', data=payload) print(p.text) req = s.get(blacklist_lolz) print(req.text) soup = beautifulsoup(req.content) divs_all = soup.find_all('a', class_='PreviewTooltip ') links_all = [div['href'] for div in divs_all] for link in links_all: number_list = [] link_page = s.get(main_lolz + link, headers=header) link_soup = beautifulsoup(link_page.content) # Some code ...
by

Related questions

0 like 0 dislike
1 answer
asked Sep 9, 2019 by serj2000
0 like 0 dislike
1 answer
asked Aug 18, 2019 by serj2000
0 like 0 dislike
1 answer
asked Jul 11, 2019 by serj2000
0 like 0 dislike
1 answer
asked Jul 16, 2019 by serj2000
0 like 0 dislike
1 answer
110,608 questions
257,186 answers
0 comments
35,405 users