Crawlab Documentation

# CLI

The CLI tools allow users to easily manage Crawlab and perform common actions including uploading spiders. It is written in Python and very easy to install.

# Install

Crawlab CLI tools is integrated with Crawlab SDK (opens new window). You can install Crawlab CLI tools by executing the command below.

pip install crawlab-sdk==0.6.b20211224-1500
1

WARNING

Currently only beta version is available, so we have to install by specifying the version.

Now you are able to use CLI tools to programmatically interact with Crawlab.

# Login

WARNING

You MUST login to Crawlab with CLI tools before performing any other actions.

You can login to Crawlab with CLI tools by executing the command below.

crawlab-cli login \
  --api_address <api_address> \
  --username <username> \
  --password <password>
1
2
3
4
  • --api_address or -a is the API address of Crawlab. Default: http://localhost:8080/api.
  • --username or -u is the username of login. Default: admin.
  • --password or -p is the password of login. Default: admin.

# Upload

You can upload spiders using CLI tools.

# Upload a new spider

If you would like to upload a new spider, you can execute the command below.

crawlab-cli upload \
  --dir <target_spider_dir> \
  [--name <spider_name>] \
  [--col_name <results_collection_name>] \
  --create
1
2
3
4
5
  • --create or -c is whether to create a new spider. If unspecified, --id or -i must be specified.
  • --dir or -d is the parameter of the spider directory to upload. Default: current working directory.
  • --name or -n is the name of the spider to create. Default: directory name.
  • --col_name or -C is the result collection name of the spider to create. Default: results_<spider_name>.
  • --cmd or -m is the execute command of the spider to create. Default: echo "hello crawlab".

# Upload to an existing spider

If you would like to upload to an exiting spider, you can execute the command below.

crawlab-cli upload \
  --dir <target_spider_dir> \
  --id <spider_id>
1
2
3
  • --dir or -d is the parameter of the spider directory to upload. Default: current working directory.
  • --id or -i is the id of the spider to upload.

# Config

You can view and set/unset config with CLI tools.

# View config

crawlab-cli config
1

# Set config

crawlab-cli config --set <key>=<value>
1
  • --set or -s is the key-value pair to set in the config.

# Unset config

crawlab-cli config --unset <key>
1
  • --set or -s is the key to unset from the config.

← Introduction →