0.介绍

scrapyd是运行scrapy的服务器.

1.scrapyd客户端和服务端的安装

1.1服务端安装

执行如下命令安装

pip3 install scrapyd

安装完成后会有个scrapyd命令,执行这个命令

(venv) ╭─lzq@localhost.localdomain ~/PycharmProjects/python_reptiles/teach/scrapy/basic/example/example  
╰─➤  scrapyd
2019-08-23T07:58:59+0800 [-] Loading /home/lzq/PycharmProjects/python_reptiles/venv/lib/python3.6/site-packages/scrapyd/txapp.py...
2019-08-23T07:59:00+0800 [-] Scrapyd web console available at http://127.0.0.1:6800/
2019-08-23T07:59:00+0800 [-] Loaded.
2019-08-23T07:59:00+0800 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 16.6.0 (/home/lzq/PycharmProjects/python_reptiles/venv/bin/python 3.6.1) starting up.
2019-08-23T07:59:00+0800 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.
2019-08-23T07:59:00+0800 [-] Site starting on 6800
2019-08-23T07:59:00+0800 [twisted.web.server.Site#info] Starting factory <twisted.web.server.Site object at 0x7f21c1648710>
2019-08-23T07:59:00+0800 [Launcher] Scrapyd 1.2.1 started: max_proc=32, runner='scrapyd.runner'
2019-08-23T07:59:13+0800 [twisted.python.log#info] "127.0.0.1" - - [22/Aug/2019:23:59:13 +0000] "GET / HTTP/1.1" 200 706 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36"
2019-08-23T07:59:14+0800 [twisted.python.log#info] "127.0.0.1" - - [22/Aug/2019:23:59:13 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "http://127.0.0.1:6800/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36"
2019-08-23T08:04:34+0800 [twisted.python.log#info] "127.0.0.1" - - [23/Aug/2019:00:04:33 +0000] "GET / HTTP/1.1" 200 706 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36"
^C2019-08-23T08:04:57+0800 [-] Received SIGINT, shutting down.
2019-08-23T08:04:57+0800 [-] (TCP Port 6800 Closed)
2019-08-23T08:04:57+0800 [twisted.web.server.Site#info] Stopping factory <twisted.web.server.Site object at 0x7f21c1648710>
2019-08-23T08:04:57+0800 [-] Main loop terminated.
2019-08-23T08:04:57+0800 [twisted.scripts._twistd_unix.UnixAppLogger#info] Server Shut Down.

打开 http://127.0.0.1:6800/

1.2服务端安装

执行安装客户端命令

pip install scrapy-client

安装完成后如果有scrapyd-deploy这个命令表示安装成功