Convey

Swiss knife for mutual conversion of the web related data types, like base64 or outputs of the programs whois, dig, curl. A convenient way to quickly gather all meaningful information or to process large files that might freeze your spreadsheet processor.

Any input is accepted:

• if a single value input is detected, all meaningful information is fetched
• multiline base64/quoted_printable string gets decoded
• log/XLS/XLSX/ODS file converted to CSV
• CSV file (any delimiter, header or pandoc table format) performs one or more actions
  1. Pick, delete or sort columns (if only some columns are needed)
  2. Add a column (computes one field from another – see computing fields)
  3. Filter (keep/discard rows with specific values, no duplicates)
  4. Split by a column (produce separate files instead of single file; these can then be sent by generic SMTP or through OTRS)
  5. Change CSV dialect (change delimiter or quoting character, remove header)
  6. Aggregate (count grouped by a column, sum...)
  7. Merge (join other file)

$ convey example.com    # single value: all meaningful information is fetched
$ convey file.csv       # CSV/log/XLS file: pick, add, filter, split, aggregate, merge columns
$ convey --server       # the same engine as a web service

pip3 install convey

Python3.10+ required.

Documentation

• Installation — prerequisites, PyPI/GitHub install, bash completion.
• Modes:
  • Single value — whois, DNS, web, encodings, regex, units, countries.
  • CSV processing — pick, add, filter, split, change dialect; caching, headless runs.
  • Aggregate & merge
  • Web service
• Fields:
  • Computing fields — field syntax, reference of all fields, whois module.
  • External fields — custom Python methods, PickMethod and PickInput decorators.
• Sending e-mails — mailing split chunks via SMTP or OTRS, dynamic templates, sending images.
• Configuration — convey.yaml, daemon, files and logs.
• CLI options
• Changelog

Brought by CSIRT.cz.