wget을 사용하여 PHP 웹사이트에서 PDF 및 zip 파일을 다운로드하세요.

Question

컬 요청은 다음과 같습니다(브라우저 개발자 도구의 네트워크 섹션에서 얻을 수 있음).

curl 'https://www.concours-maths-cpge.fr/' -H 'Cookie: startBAK=xxxxxxxxxxxxxxxx; start=xxxxxxxxxxxx' -H 'Origin: https://www.concours-maths-cpge.fr' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.8' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.86 Safari/537.36' -H 'Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryBj169hPosWDbuaq7' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'Cache-Control: max-age=0' -H 'Referer: https://www.concours-maths-cpge.fr/' -H 'Connection: keep-alive' --data-binary $'------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="cmd"\r\n\r\n\r\n------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="mode"\r\n\r\nsimple\r\n------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="annee"\r\n\r\n2017\r\n------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="concours"\r\n\r\n0\r\n------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="filiere"\r\n\r\n0\r\n------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="matiere"\r\n\r\n0\r\n------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="epreuve"\r\n\r\n0\r\n------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="commande"\r\n\r\nrechercher\r\n------WebKitFormBoundaryBj169hPosWDbuaq7--\r\n' --compressed

컬링 후 데이터는 다음과 같습니다.

<head><title>Sujets de concours</title>
<link rel='stylesheet' href='javascript/table.css' type='text/css' />
<script language='javascript' src='javascript/table.js'></script>
</head>
<body background='images/julia.gif'>
<div align=left><table style='background:none'><tr>
  <td><img src='images/ups.gif'>
  <td><h1>Union des Professeurs de classes préparatoires Scientifiques<br>Sujets de concours
</tr></table></div>
<div align='center'><form enctype='multipart/form-data' method='post'>
<input type='hidden' name='cmd' value=''/>
<input type='hidden' name='mode' value='simple'/>
<input type='hidden' name='annee'       value='2017' />
<input type='hidden' name='ordre'       value='-annee:+nom:+filiere:+matiere:+epreuve' />
<table border=2 rules='none' style='table-layout:fixed' width='90%'><tr><td><button title='tri croissant' style='width:100%' onclick='mysubmit(this,"commande=rechercher|ord=+annee")'>année</button><td><button title='placer à gauche' style='width:100%' onclick='mysubmit(this,"commande=rechercher|ord=+nom")'>concours</button><td><button title='placer à gauche' style='width:100%' onclick='mysubmit(this,"commande=rechercher|ord=+filiere")'>filière</button><td><button title='placer à gauche' style='width:100%' onclick='mysubmit(this,"commande=rechercher|ord=+matiere")'>matière</button><td><button title='placer à gauche' style='width:100%' onclick='mysubmit(this,"commande=rechercher|ord=+epreuve")'>épreuve</button><td style='width:5em; padding:0px'></tr>
  <tr class='even'><td class='rowspan' rowspan=10><a title='2017'>2017</a><td class='rowspan' rowspan=2><a title='Agrégation externe'>Agr-ext</a><td class='rowspan' rowspan=2><a title='M1'>M1</a><td class='rowspan' rowspan=2><a title='maths'>maths</a><td><a title='algèbre'>algèbre</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17a4a")'>fichiers</button></tr>
  <tr class='odd'><td><a title='analyse'>analyse</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17a4y")'>fichiers</button></tr>
  <tr class='even'><td><a title='Agrégation concours spécial'>Agr-special</a><td><a title='autre'>autre</a><td><a title='maths'>maths</a><td><a title='unique'>unique</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17azu")'>fichiers</button></tr>
  <tr class='odd'><td class='rowspan' rowspan=2><a title='Agrégation interne'>Agr-int</a><td class='rowspan' rowspan=2><a title='autre'>autre</a><td class='rowspan' rowspan=2><a title='maths'>maths</a><td><a title='1'>1</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17bz1")'>fichiers</button></tr>
  <tr class='even'><td><a title='2'>2</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17bz2")'>fichiers</button></tr>
  <tr class='odd'><td class='rowspan' rowspan=2><a title='Banque Agro-Véto'>Agro</a><td class='rowspan' rowspan=2><a title='BCPST'>BCPST</a><td class='rowspan' rowspan=2><a title='maths'>maths</a><td><a title='A'>A</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17ab1")'>fichiers</button></tr>
  <tr class='even'><td><a title='B'>B</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17ab2")'>fichiers</button></tr>
  <tr class='odd'><td class='rowspan' rowspan=3><a title='Concours Commun Polytechniques'>CCP</a><td class='rowspan' rowspan=3><a title='MP'>MP</a><td><a title='info'>info</a><td><a title='option'>option</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=i17pmo")'>fichiers</button></tr>
  <tr class='even'><td class='rowspan' rowspan=2><a title='maths'>maths</a><td><a title='1'>1</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17pm1")'>fichiers</button></tr>
  <tr class='odd'><td><a title='2'>2</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17pm2")'>fichiers</button></tr>
<tr><td colspan=5><table width='100%'><tr>
  <td>sujets 1 à 10 (total = 63)
  <td align='right'><button title='page suivante'   onclick='mysubmit(this,"commande=rechercher|init=10")'>&gt;</button></tr></table>
  <td><button title='modifier la recherche' style='width:100%' onclick='mysubmit(this,"commande=connexion")'>retour</button>
</tr></table>
</form></div></body>

그리고 wget을 사용하는 PDF 링크가 없습니다. 따라서 ajax를 사용하면 클릭 시 pdf 링크를 생성하는 데 javascript가 사용됩니다.

유일한 해결책은셀레늄 네트워크 드라이버기본적으로 브라우저의 상호 작용을 자동화합니다. Selenium과 함께 chrome/chormium 또는 firefox를 사용할 수 있습니다(코드가 Selenium과 함께 실행되면 열린 브라우저 창이 자동으로 클릭을 수행함). 그러나 가장 좋은 방법은 헤드리스 브라우저를 사용하는 것입니다.PhantomJS셀레늄으로.

다음은 Selenium(및 PhantomJS)을 사용한 스크래핑에 대한 몇 가지 예제 링크와 몇 가지 질문입니다.

Python Selenium 및 PhantomJS를 사용하여 스크래핑

Python 및 PhantomJS를 사용한 헤드리스 Selen 테스트

WebDriver click() 및 JavaScript click()

Python 및 PhantomJS와 함께 Selenium을 사용하여 파일 시스템에 파일 다운로드

Answer 1