openpolitics.com

How to save “complete webpage” not just basic html using Python

You can download a complete webpage with the pywebcopy library:

<span class="kwd">from</span><span class="pln"> pywebcopy </span><span class="kwd">import</span><span class="pln"> save_webpage

url </span><span class="pun">=</span> <span class="str">'http://some-site.com/some-page.html'</span><span class="pln">
download_folder </span><span class="pun">=</span> <span class="str">'/path/to/downloads/'</span><span class="pln">    

kwargs </span><span class="pun">=</span> <span class="pun">{</span><span class="str">'bypass_robots'</span><span class="pun">:</span> <span class="kwd">True</span><span class="pun">,</span> <span class="str">'project_name'</span><span class="pun">:</span> <span class="str">'recognisable-name'</span><span class="pun">}</span><span class="pln">

save_webpage</span><span class="pun">(</span><span class="pln">url</span><span class="pun">,</span><span class="pln"> download_folder</span><span class="pun">,</span> <span class="pun">**</span><span class="pln">kwargs</span><span class="pun">)</span>

The CiteIt.net plugin converts html to raw text, which results in the formatting being lost.