enh: convey.log moved from the current directory to the user log dir (~/.cache/convey/log/), created only when something is logged, rotated; --log-file to change it; an unwritable directory no longer prevents convey from starting
fix: multiple merges no longer all use the last merged file
fix: tests will not flood up CWD
fix: file cache got lost (saved as null) after an aggregation or a custom output dialect, and the next run crashed on it; a CLI aggregation re-run over the cached file is no longer added twice
fix: columns sharing a name (ex: after a merge) were collapsed into one in the column dialogs (aggregate, split, pick/delete columns, new column source) – the first one pointed to the last column and "Pick or delete columns" silently dropped the rest
feat: RDAP support – registries are asked over RDAP first, whois stays as a fallback (--whois.backend auto|rdap|whois); the ASN missing in RDAP is resolved from the Cymru DNS zone (--whois.asn-lookup)
fix(security): daemon socket moved from the world-reachable /tmp/convey_socket to a private per-user directory (prevents a local user from capturing convey's argv, spoofing its output, or driving the daemon)
fix(security): cache files can no longer execute code on load (a tampered cache used to be a deserialization RCE via jsonpickle)
enh: whois cache stored as plain JSON – ~10× faster to write, ~3× to read, ~⅓ smaller
template function {{ print_attachment() }} renamed to {{ attachment() }}
aggregated results will provide valid CSV, not fancy but hardly parsable table
sending works with SMTP or OTRS6 (OTRS3 is deprecated now)
removed --csirt-incident macro (easily replaced by other CLI flags)
webservice
uwsgi file (HTTPS support)
clear=web parameter
able to parse pandoc simple table format (headers underlined with ---------) and a lot of spaces between columns
local_country is empty by default (you do not have to be a national CSIRT to use this program)
program help flags grouped
types
csirt_contact now may be derived from tld
web module now ignores invalid HTTPS certificates
form_names prints out more information about tags without name attribute
base64 no more limited to UTF-8, better detection
phone formats not confused with dates
ip -> url conversion generates a proper PTR hostname
wrong_url protocol case insensitive (accepts "hXXp")
Unix time is reasonably recognized as a timestamp
cc_contact converts abusemail to value given by contacts_cc.csv
dialog
edit default e-mail templates from config submenu
delimiter now may contain tab character, input by "\t" or "tab"
show current SMTP server in the send submenu
--config now can edit any config file, not only config.ini
print out settings so that it can be reprocessed via bash command (currently experimental, on file exit)
displays what is happening if WHOIS cache is being loaded/saved longer than 1 s
atomic thread safe information printing – no more blinking and shuffled info; few records stay on the screen when information refresh
statistics info reworded
reset menu renamed to redo
sending menu – testing is the default option (you do not want to send all by mistake too early) and attachments are sorted alphabetically
positional vs keyword arguments order resilience
flags
--whois-delete-unknown
--whois-reprocessable-unknown
--whois-cache
--web-timeout
--subject, --body (even with BASE64), --references
when using --body flag, combine the text with the template
selecting column by negative numbers to match the last added column (--field tld,-1)
while splitting --output left blank prints out the file contents to the STDOUT after the processing
fix: possible to split by a mere type (exact column ID/name not needed)
internal
OTRS no more appends ".txt" to the attachment name
web scraping bug of suspicious form-tag without name attribute present
fix: print all attachments flush
fix: cache empty whois prefix (ex: of a wrongly formatted IP)
whois cache saved only when changed (useful when having a huge cache)
mitigation for another type of a wrong WHOIS response
fix: reprocessing while changing dialect
fix: catch preview exception
fix: threading atomic write
applying filter before line processing if possible (spares a lot of time, ex: skip lines before WHOIS processing)
fix Python3.6: disabled logging messages from daemon (threw errors)
fix: processing velocity info (lines / s) showed garbage since threads implemented
fix: aggregate via Alt+a from main menu + any after-processing dialog (like resolve unknowns) caused the terminal to freeze (cannot use input() while another prompt toolkit session is active)
CLI better source_type determining
web module boosts and fixes
fix: when reprocessing (ex: invalid lines) after loading, file will get cleared no more
fix: CIDR when WHOIS returns directly CIDR instead of prefix. Currently, we do not guarantee prefix will be in the form "... - ...", it may has directly CIDR form if WHOIS tells us so
fix 1.3.6: file name duplicate when reprocessing
fix: aggregation can be disabled even if processed before
NTFS compatible abroad attachment names. Instead of a colon we use double at-sign to delimit country code from the chosen e-mail when splitting by an incident-contact field.
fix: PickBase used from CLI through --external
fix: empty convey.log is no more created in the directory when writing python3 -m [LETTER].[TAB] into terminal
unmaintained library validate_email replaced by py3-validate-email
for non-local countries produces now the file whose name is in the form "abroad:abusemail@example.com"
when non-local country is missing from contacts_abroad, it will be sent to its abusemail contact (but preserving mail_template_abroad)
attachment name no more prepended with prefix "part-"
by default, produced file CSV dialect is set to standard: comma, double quotes. (This may be changed in the config.ini by letting the relevant fields in CSV section blank to let the output file have the same dialect as the input.)
flags
--aggregate – count grouped by a column, sum, etc.
--daemon, daemonize – since it takes around 0.75 s to start the program, notably because of the external libraries (pint, bs4, requests), when doing a single query or a headless processing (no menu involved) we may reduce this time at least ten times with a daemon)
--type – specify type of the given column(s), useful when treating a column that cannot be easily detected (as country_name)
--output left BLANK causes output be piped to STDOUT instead of to a file.
--reprocess
--testing
--send, --send-test [e-mail] [template]
--jinja
--attach-files
--delimiter-output, --quote-char-output, --header-output (may be set to false) to change output file syntax
LICENSE included
emergency input mode – when piping into the program (instead of giving the input as an argument), convey tries to gain a reduced STDIN from the terminal process #38
bash completion
webservice accepts field and type arguments (same as CLI flags)
fix
adding external modules at runtime via menu
Ctrl+C works when interrupting wizzard as expected #39
detect multiline quoted_printable
better phone format recognition
multiple STDIN processing in a single session
CSV flags (like --header) when processing in --yes mode
web relative redirect working
max number of redirects is 10
type: form_names
interface
nicer output when processing from STDIN
aggregation more human readable
exclude filter + filter menu (composed from unique, exclude and include filter)
remove some of the processing settings
internal
daemon pipes stderr as well
got rid of python-Levenshtein package and did not use psutil package so installing whole gcc is not needed anymore
after resetting settings fields keep their auto-detection information
tests
sending: huge improvement of the interface
SMTP sending uses the envelope library
jinja templates, including helper instruments: print_attachment, amount, row, joined, first_row
fix: new installation should now correctly place contacts files in .config
fix: refresh abroad contact list e-mails when restart even if the file has been processed before
PyPi installer requirements fix
Whois module
won't throw error if host can't be resolved
huge refactoring (may lead to the behaviour when Country is taken from ARIN and Netname from AfriNIC if AfriNIC points us to ARIN that doesn't state Netname)
thousands of unique prefixes tried, many uses cases handled
naive database of country names, so that country may be guessed from non-standardised "address" field
some well known erroneous LIR responses are re-requested by their respective RIR
CIDR with host bits set ( = invalid network) translates to IP when asking whois (RIR would return 'invalid search key')
CIDR translated to an IP before asking whois
number of discovered prefixes should during processing displayed, as well as real whois server URLs
LACNIC quota exceeded marks the rows to be re-queued
sets internal process locale to en_US so that we can grep the same result at environments using different language
processing:
logs in format "time IP src port > IP dst port" can be automatically transformed to CSV before processing
multiline base64 and quoted_printable strings (seen in e-mails) may be input → automatically decoded
methods may return a list (the row will be duplicated)
implicitly convert to plaintext if possible when adding new column (ex: base64 will be implicitly decoded before its value being passed to a new reg column)
decorator PickMethod, PickInput to allow the user choose the way generating will work
internal
os replaced by pathlib
Config.getboolean merged to Config.get
Identifier (former CSVGuesses) is now object oriented, no more spaghetti
huge refactoring
fix: log files
Processor handles Whois statistics no more, so Identifier Whois methods do not return tuple anymore (standardised)
advanced header detection
interface
able to automatically add/remove config file flags at program upgrade if allowed by user
when no known method is known to process a field, an example is given
when more than 9 options, you can use letters as shortcuts
main screen shows colorized result, preferable in the form of table if the terminal is wide enough
new 'code' field type for writing arbitrary code
when not in debug mode, a GitHub issue is filled out automatically at crash
autoopen_editor opens when analysis starts but now only if splitting by a column
Config file startup integrity check: missing items and sections may be inserted automatically after an upgrade
fix: do not reprocess file if moved to another location
fix: empty file check
fix: dialect ignored when generating a sample, whois inconsistency toughness
fix: when config file is a symlink and is broken, do not confuse user with creating a new set of config file I.E. in .local/bin. Instead, demand mounting the symlink (or exit or recreate files).
fix: whois fetching asn + netname again
'/' is forbidden char in linux file names, when splitting by IP prefix, the char is replaced with a dash
reading from stdin instead of a file
when stdin input is a single value, auto-detect it, compute all possible results and quit (use case: inputting a base64 string will decode it, IP will produce all whois information table)
new file_or_input config flag
new save_stdin_output config flag
Config file startup integrity check (you might have missing items since last upgrades)
dropped default_dir and default_file config items & functionality. I doubt anyone has ever used that. These were to store the incidents directory for the case we did not provide an exact filename to treat when launching convey.
Adding a column: if there is a single column we can source from, skip the dialog
Since delimited, quote char and header guesses are mostly right, I stripped two dialog questions to a single one
Main menu uses "←←←←←" symbol for hinting default value that is triggered when user hits Enter. When you select a processing action you don't have to write "p" for "process" anymore.
CIDR type
varying number of columns exception check
when analyzing, show number of file descriptors currently being written
unmaintained library lepl replaced by validate_email
it worked somehow but my colleague faced a strange issue in Whois module. Match object couldn't be called as a dictionary. I stepped out the change to see that Python3.5 support was dropped since 18.8., 51d7a90 , just after the first mature 1.0.0 release.
logging while processing invalid rows write whole traceback
file sizes in "show all details"
csirtmail field fix, creating Config files: wrong path might be reported …