Converting Keep to Pet Snippets

I've been wanting to move from OrkoHunter's Keep to the more feature rich knqyf263's Pet but kept procrastinating because I already had hundreds of snippets in Keep and dreaded the copy and paste nightmare to move them one by one.

Here's a quick bash script I wrote to parse through Keep's .json'ish format and convert to Pet's TOML format. Hopefully this helps someone else facing the same issue.

#! /bin/bash

###Leif Gregory <leif@devtek.org>
###Copy Keep's commands.json file to the same folder as this script and run it.
###It will output a file in the same folder called snippet.toml. Make a backup
###copy of your old snippet.toml and move the new one into its place. Run
###"pet list" and hopefully you'll get a clean output. If not, pet will tell
###you which line of snippet.toml the error is on. Compare it with the original
###commands.json file.

#Usage [scriptname]
#e.g. ./keep2pet

inputfile=./commands.json  #Typically found in $HOME/.keep/
outputfile=./snippet.toml  #Move this file to, typically, $HOME/.config/pet/

#Keep's commands.json starts with { and ends with } which the first and last
#sed take care of. The 2nd sed breaks up the single line with all commands
#and descriptions into one command and description per line. The delimiter is
#unfortunately a comma and I had lots of commas in my descriptions. So, it
#breaks on '", "' which means it removes the leading and trailing double quotes.
#The 3rd and 4th sed fixes those. Lastly, it writes all these lines to a temp
#file which gets removed at the end. 

cat $inputfile | sed 's/^{\"//g' | sed 's/\", \"/\n/g' | sed 's/^/"/g' | sed 's/$/"/g' | sed 's/\"}$//g' > ./keep2pet-temp

while read -r line
do
    #Now we're down to one command and description delimited by ": "
    command=$(awk -F ": " '{print $1}' <<< $line)
    description=$(awk -F ": " '{print $2}' <<< $line)

    #Write each commands.json snippet to TOML format file
    echo "
[[snippets]]
  description = $description
  command = $command
  output = \"\"" >> "$outputfile"
done < keep2pet-temp

rm ./keep2pet-temp 

Summing numbers at the end of lines in a text file

I wrote this one liner to sum the counts of various attacks that were being blocked and logged by the firewall during an automated scan.

Let's say your log looks something like this.

Contents of log.txt

Microsoft Windows win.ini Access Attempt Detected 30851 vulnerability 782 
HTTP Cross Site Scripting Attempt 32658 vulnerability 288
Generic HTTP Cross Site Scripting Attempt 31475 vulnerability 94
HTTP /etc/passwd Access Attempt 35107 vulnerability 82
HTTP SQL Injection Attempt 30514 vulnerability 52
PHP CGI Query String Parameter Handling Information Disclosure Vulnerability 34804 vulnerability 28
Generic HTTP Cross Site Scripting Attempt 31476 vulnerability 24
Apache Tomcat URIencoding Directory Traversal Vulnerability 35298 vulnerability 13
Export RSA cipher suite detected 37493 vulnerability 11
HTTP SQL Injection Attempt 33338 vulnerability 10
Squid HTTP Header Parsing Assertion Failure Denial of Service Vulnerability 39682 vulnerability 10
Oracle 9i Application Server Dynamic Monitoring Services Anonymous Access 33756 vulnerability 8
HTTP SQL Injection Attempt 35823 vulnerability 6
PHP-Charts PHP Code Execution Vulnerability 37008 vulnerability 6
Microsoft Internet Explorer Cached Objects Zone Bypass Vulnerability 33813 vulnerability 4
Advantech Studio NTWebServer Arbitrary File Access Vulnerability 35784 vulnerability 2
Generic HTTP Cross Site Scripting Attempt 30847 vulnerability 2
Microsoft IIS ServerVariables_JScript. asp Information Disclosure 33073 vulnerability 2
Microsoft IIS 5.0 Form_JScript.asp XSS Vulnerability 32775 vulnerability 2
Joomla HTTP User Agent Object Injection Vulnerability 38519 vulnerability 1
OpenSSL Status Extension Memory Leak Denial of Service Vulnerability 39926 vulnerability 1




The five digit numbers before the word "vulnerability" are an ID and the digits at the end of each line are the counts of how many were blocked. We could sit here with a calculator and add all the digits at the end together, but the below one liner will do it for you.

grep -oP '\d{1,4}$' log.txt | xargs | tr ' ' + | bc

When run, it will output: 1428

Let's break down the one liner to understand what each part is doing.

grep -oP

• -o tells grep to only output the matched text, not the whole line. 
• -P tells grep we want to use PCRE regexp vs the default POSIX regexp.

'\d{1,4}$'

• \d tells grep we're looking for a digit.
• {1,4} tells grep we're looking for a number that will be between one and four digits.
• $ tells grep we're looking for these one to four digit numbers to be at the end of the line.

Technical note: Just in case someone has an issue with how I phrased the above, what we're really telling grep is to look for between one and four digits sequentially.


log.txt is the name of the log file

| means we're piping the output of the previous command, grep in this case, into another command

xargs is going to take all the one to four digit numbers found and concatenate them into a string like this:

782 288 94 82 52 28 24 13 11 10 10 8 6 6 4 2 2 2 2 1 1

tr ' ' + will replace all the spaces between the numbers with the plus sign like this:

782+288+94+82+52+28+24+13+11+10+10+8+6+6+4+2+2+2+2+1+1

bc is a command line calculator that will evaluate the string provided by tr above and produce a sum. 1428 in this case.

If you needed to subtract instead of add, you'd just change the '+' in the tr command to '-'. Or if your counts could be a five digit number just change '{1,4}' to '{1,5}'. Or let's say your counts will always at least be three digits, but no more than five, you'd change it to '{3,5}'