Python emulates web-browser in Babel
Preamble
From the official repository
Spynner is a stateful programmatic web browser module for Python. It is based upon PyQT and WebKit. It supports Javascript, AJAX, and every other technology that !WebKit is able to handle (Flash, SVG, …). Spynner takes advantage of JQuery. a powerful Javascript library that makes the interaction with pages and event simulation really easy.
Using Spynner you would able to simulate a web browser with no GUI (though a browsing window can be opened for debugging purposes), so it may be used to implement crawlers or acceptance testing tools.
See usage here
Installation
Dependencies
Activate your virtual environment
The main goal of this article is to install spynner with Python3.
Check up current Python version:
python --version
Python 3.4.3
Now we'll install spynner and its dependencies into your venv
SIP
cd ../soft/ wget https://sourceforge.net/projects/pyqt/files/sip/sip-4.18.1/sip-4.18.1.tar.gz tar -xvf sip-4.18.1.tar.gz cd sip-4.18.1/ python configure.py make make install
PyQt
cd ../soft/ wget http://sourceforge.net/projects/pyqt/files/PyQt4/PyQt-4.11.4/PyQt-x11-gpl-4.11.4.tar.gz tar -xvf PyQt-x11-gpl-4.11.4.tar.gz cd PyQt-x11-gpl-4.11.4/ python configure-ng.py make make install
Test it up
from PyQt4.QtGui import QImage
If you successfully evaluate [by C-c C-c=] =import above -
take my congratulations:
PyQt4 was istalled
Build and install custom autopy for python3
git clone git://github.com/msanders/autopy.git
cd autopy/
python setup.py build
python setup.py install
Pyside
sudo apt-get install cmake pip install pyside
Libxml2?
Spynner installation
Clone the repository
cd /usr/local/share/DVCS/
git clone https://github.com/makinacorpus/spynner.git
Switch on py3 branch
cd /usr/local/share/DVCS/spynner
git checkout py3
git branch -a
Already on 'py3' M examples/webkit_methods.py M run_tests.sh Your branch is up-to-date with 'origin/py3'. master * py3 remotes/origin/HEAD -> origin/master remotes/origin/master remotes/origin/py3
Review the content
cd ../spynner
ls
| bootstrap.py |
| b.sh |
| build |
| buildout.cfg |
| CHANGES.rst |
| dist |
| examples |
| MANIFEST.in |
| minitage-buildout.cfg |
| minitage.cfg |
| minitage-test.cfg |
| README.rst |
| run_tests.sh |
| setup.cfg |
| setup.py |
| src |
| test |
Install spynner
python setup.py install
Test spynner in general
import spynner import pyquery browser = spynner.Browser(debug_level=spynner.DEBUG) browser.create_webview() browser.show() spynner.Browser()
Create the executable file
from time import sleep from spynner import browser br = browser.Browser( # debug_level=4 ) br.load('http://pypi.python.org/pypi') br.create_webview() br.show() br.wk_fill('input[id=term]', 'spynner') br.wk_click("input[id=submit]", wait_load=True, timeout=5) print("Noticed the search") sleep(3) anchors = br.webframe.findAllElements('#menu ul.level-two a') anchor = [a for a in anchors if 'Browse' in a.toPlainText()][0] br.wk_click_element_link(anchor, timeout=10) print("Noticed the click on the browse") sleep(3)
Run script without X11
xvfb-run python spynner/myscript.py
Noticed the search Noticed the click on the browse
blog comments powered by Disqus