Exocortex tools part I: social media automation with very small shell scripts
Exocortex tools part I: social media automation with very small shell scripts
Exocortex Tools is a series describing my personal set of utility shell scripts. The term ‘Exocortex tool’ in this context comes from @VirtualAdept, but this does not represent his toolchain. This entry will be focusing on post, links.sh, and notes.sh.
There’s a lot of interest in cross-posting tools & alternatives to traditional web-based centralized social media lately. I have been using a toolchain I wrote myself for several years, and I have avoided describing it until now because the tools I wrote are so simple that I didn’t consider them worth describing. However, I’ve seen a lot of people using webtech or otherwise doing much more work than necessary, while failing to achieve feature parity with what I’ve got.
When it comes to any form of automation, the unix shell is your friend: it will allow you to compose existing tools together with a minimum of fuss, and existing tools are typically pretty well-suited to using it (or else programs like curl are well-suited to bridge that gap).
In my particular case, I have a presence on a variety of social networking sites, only a few of which have IFTTT integration, and I would like to automate the broadcast of certain types of posts. In particular, I want the ability to post arbitrary microblogs to all services simultaneously, the ability to post links with automatically-fetched titles, and to have those links show up in a pinboard-like minimal interface on my website. Furthermore, I would like integration with a system for short plaintext notes I already keep.
The easiest problem to solve is the problem of simultaneous broadcast. There is a command-line tool for posting to almost every social network. In my case, I use the ruby gem ‘t’ for twitter, the pip package ‘tootstream’ for the fediverse, the npm package ‘sbot’ for secure scuttlebutt, and the twtxt command line client for twtxt, while relying upon IFTTT to relay twitter posts elsewhere (since twitter is the only microblogging service mentioned above that IFTTT knows about). The shell script is VERY small.
#!/usr/bin/env zsh
args="$@"
t post "$args"
yes | twtxt tweet "$args"
echo -e "toot -v $args\nu" | tootstream
sbot publish --type post --text "$args"
This tool is named ‘post’ & it does what it says on the tin: it posts the args (quoted or otherwise) everywhere I care about. Posts that are too long for twitter will fail to post there and merely be posted to the other services (which have larger maximum post sizes).
As for links, the problem is slightly more difficult. We need a format for storing link information permanently, a mechanism to transform the list of links into a web page, and a mechanism to add a link to the list.
I decided to use a three-column TSV to store link information. I append lines to the end, so the resulting file is in chronological order. The columns are: URL, time, and (optional) title.
Producing a static HTML file from this is straightforward: we produce the beginning and end of the HTML file, then (reading the list of links backwards) produce an entry for each.
function fmtlinks() {
echo "<html>"
echo '<head><title>Links</title><link rel="stylesheet" type="text/css" href="vt240.css"></head>'
echo "<body>"
echo "<table>"
echo "<tr><th>Link</th><th>Date</th></tr>"
tac ~/.linkit|
awk '
BEGIN{FS="\t"}
{
title=$1;
if($3) title=$3;
print "<tr><td><a href=\"" $1 "\">" title "</a></td><td>" $2 "</td></tr>"
}'
echo "</table>"
echo "<center><a href=\"https://github.com/enkiv2/misc/blob/master/links.sh\">Generated with links.sh</a></center>"
echo "</body>"
echo "</html>"
}The only logic in the center is to use the URL as the title if the title entry is empty. This is a small & simple script, and could be made shorter if I sacrificed readability. Because HTML is a nightmare, I have erred on the side of clarity over terseness.
The most complicated part of adding links is fetching their title, and the most complicated part of fetching link titles is dealing with strange nonstandard HTML escape logic. Here is my title-fetching code:
function getTitle() {
curl "$1"| grep -a -i "<title>" | head -n 1 |
sed 's/^.*<[tT][iI][tT][lL][eE]>//;s/<\/[tT][iI][tT][lL][eE]>.*//' |
sed 's/'\;/'"'"'/g;s/'\;/'"'"'/g;s/"\;/"/g' |
tr '\221\222\223\224\226\227' '\047\047""--'
}First I pull down the HTML into a pipe, so the fetch for a large page will actually be aborted once I have found the first title tag, then I remove everything before the end of the open tag and everything after the end of the close tag. After that, all of the logic is related to processing common HTML escapes and stripping commonly-found but invalid characters (like ‘smartquotes’).
Adding a link involves a few steps: we must get the title, create a truncated title so that both title and URL will fit on twitter, and post the result (if it exists). I additionally use Vice Motherboard’s “mass_archive” script to add any URL in my link archive to the wayback machine and archive.is.
function linkit() {
(which mass_archive 2>&1 > /dev/null && mass_archive "$1")
export LC_ALL=en_US.UTF-8
echo -e "$1\t$(date)\t$(getTitle "$1")" >> ~/.linkit
post="$(
export LC_ALL=en_US.UTF-8 ;
tail -n 1 ~/.linkit | awk '
BEGIN{FS="\t"}
{
url=$1 ;
title=$3 ;
if(url!=title && title!=""&&title!=" ") {
if(length(url)+length(title)>=280) {
delta=(length(url)+length(title))-280;
delta+=4;
if(delta<length(title)) {
title=substr(title, 0, length(title)-delta) "..." ;
print title " " url
}
} else print title " " url
}
}' |
grep -a . | head -n 1 | recode -f UTF8)"
[[ -n "$post" ]] && post "$post"
stty sane
}We first run mass_archive if it exists in the path. Then, we write an entry to the link archive TSV. We read the final entry in that TSV, pull out the URL and title, and if the combined size of the URL and the title is more than 280 characters, we truncate only the title portion, adding an elipsis at the end. If the URL itself is longer than 280 characters, we instead emit an empty string. Once we’ve done that, we grab only the first line (in the rare case that a title will somehow emit a newline) and force the encoding to UTF8. Having done that, we post.
The final line, ‘stty sane’, exists only to clean up glitches that sometimes occur when curses-based clients (like tootstream) exit before cleanup — for instance, if I force-killed it before it had finished posting a link.
The remaining function in our link-archive system is just a helper method for upload:
function uploadlinks() {
fmtlinks > ~/index.html
scp ~/index.html $1
}My note system is just a plain text file, to which I append single lines, with a handful of helper functions. I have special post-related helper functions for ‘band name of the day’ (which I add to my notes, since I mostly use that feature for storing interesting turns of phrase) and ‘bad idea of the day’ (which I post but do not add to notes).
#!/usr/bin/env zsh
[[ -e ~/.notes ]] || touch ~/.notes
function addnote() {
read x
echo "$x" >> ~/.notes
}
function rnote() {
if [[ $# -eq 0 ]] ; then
shuf -n 1 ~/.notes
else
egrep "$@" ~/.notes | shuf -n 1
fi
}
function gnote() {
egrep "$@" ~/.notes
}
function lnote() {
if [[ $# -eq 0 ]] ; then
tail -n 1 ~/.notes
else
tail -n "$@" ~/.notes
fi
}
bnotd ()
{
echo "$@" | addnote;
post "Band name of the day: $@"
}
biotd ()
{
post "Bad idea of the day: $@"
}
Conclusion
This is not a pretty system. It’s not necessarily an efficient system. However, it’s an extremely straightforward, low-effort system that works reliably enough for its intended goals.
Since it’s an idiosyncratic system built specifically for my needs, it presents features that other users will not desire. So, I don’t recommend anyone adopt it. However, let it be seen as evidence that this kind of thing can be done with very small shell scripts, and with next to zero developer effort.
Every time I see someone implement a trivial static-site generator and then decide to use a web service to post to it, I die a little inside: why not eliminate the most irritating thing about any website (the web portion)? The answer is probably “I didn’t think of it”.
Well, now you’ve thought of it.