During the last Australian election we ran an ambitious project that tracked campaign spending and political announcements by monitoring the Facebook pages of every major party politician and candidate.
The project, dubbed the “pork-o-meter” (after the term pork-barreling), was hugely successful in being able to identify distinct patterns of spending based on vote margin, or incumbent party, with marginal electorates receiving billions of dollars more in campaign promises than other electorates.
All up, we processed 34,061 Facebook posts, 2,452 media releases, and published eight stories (eg here, here and here) in addition to an interactive feature. We also used the same Facebook data to analyse photos posted during the campaign to break down the most common types of photo ops for each party, and how things have changed since the 2016 election.
We were able to discover more than 1,600 election promises, amounting to tens of billions of dollars in potential spending. Our textual analysis later found almost 200 (112 in marginal seats) of the Coalition’s promises were explicitly conditional on their winning the election. This means much of the targeted-largesse may never have been widely known without our project.
Teasing out a few hundred election promises from millions and millions of words is like finding a needle in a haystack, and would have been otherwise impossible for our small team in such a short time frame without making use of machine learning.
Because machine learning is still something of a rarity on the reporting side of journalism (as far as I know this project is a first of its kind for the Australian media, with other ML uses mostly concentrated on content management systems and publishing), we thought it would be worthwhile to write a more in-depth article on the methods we used, and how we’d do things differently if we had the chance.
The problem (posts, lots of posts)
Commuter carparks. Sports rorts (version one and two). CCTV. Regional development grants. Colour-coded spreadsheets and great big whiteboards.
The history of elections and government funding in Australia is littered with allegations and reports outlining how both major parties have directed public money towards particular areas, whether it’s to shore up marginals or reward seats held by their own members.
However, often these reports come well after the money has been promised or awarded, following audits or detailed reporting from journalists and others.
For the 2022 election we wanted to track spending and spending promises in real-time, and keep track of how much money goes towards marginal seats, and how this compares to what each seat would receive if the funding was shared equally.
However, to do this we’d need to monitor every election announcement made by a politician, from lowly backbenchers promising $1,000 for a shed to billion-dollar pledges by party leaders. From following leader announcements in 2016, we knew that announcements could appear in media releases, local media, and on Facebook.
We decided to focus on Facebook and media releases posted on personal and party websites.
To gather the data we used the Facebook API to collate politicians’ posts into a SQLite database, and wrote web-scrapers in Python for over 200 websites to get text from media releases which was then also stored in SQLite.
The biggest challenge was then how to pull out the posts that had funding announcements in them from the rest.
The solution (machine learning and manual labour)
The output we wanted was to have a final database of only election spending promises, classified into categories, such as sport, community, crime, and so on. Each promise would also be assigned to either a single electorate or state or territory, depending on the location which would benefit most from the spending. This would allow the necessary analysis by seat and party status we’d need for news stories.
Our initial approach was to classify two weeks’ worth of Facebook posts. This initial analysis showed some commonalities in posts and releases that contained election promises. These included references to money, mentioning specific grant programs, and some keywords. But just selecting posts that contained these features would have missed a lot and had a very high false positive rate.
So we went with a blended approach. We used pre-trained language models to extract keywords, geographic locations, grant program names, named entities (like the Prime Minister), and any references to money. We then manually classified 300 randomly selected posts as either containing election promises or not. We lemmatised each word (turned them into their dictionary form, removing tense and pluralisation etc.), and turned each text into a series of numbers (a word embedding, or vector).
The vectors were created using term frequency-inverse document frequency (tf-idf), which assigns values based on how common a word is in a single text compared to the rest of the texts. This emphasises some of the differences between the texts, and together with the cosine similarity (based on the angles of the vectors if plotted), allowed us to group posts and releases that were likely about the same topic.
Finally, we trained a logistic regression model using the posts we had already manually classified. A number of other machine learning techniques were tested, but logistic regression was consistently the most accurate for our binary classification task - election promise or not.
With the classifier trained and all the extraction scripts setup, we created a pipeline where all new posts had pertinent features extracted and then a prediction was made. Any post that had a combination of features and was predicted to contain an election promise was flagged for manual review. Any media release that was dissimilar (based on cosine similarity) from the Facebook posts were similarly processed and flagged for review. We repeatedly retrained our classifier throughout the election campaign as we got more and more confirmed data.
Once the classifier was up and running, our approach was:
Scrape Facebook posts and media releases
Run posts through our classifier and duplicate checker
Manually check posts flagged as announcements and remove duplicates, add other categories and details needed
Find any media releases that were dissimilar to the Facebook posts and process them
Manually double-check all the data before publishing
Things we learned
Despite the automation, this process was still time-consuming. However, we were able to run the project in a campaign week with two days of work from two journalists and an intern working three or four days (with extra time from news and political reporters on the actual stories). Without the automation and machine-learning side of things, the same project would have required quite many more people to achieve the same result in the same time.
This was our first attempt at such a large machine learning and natural language processing project, and there’s quite a bit for us to take away and improvements that could be made.
For starters this project was almost entirely conducted using our work laptops, and so choices were made that best utilised the computer power we had available. During testing we played with more complicated methods to create word embeddings, such as Google’s BERT transformer. This would have allowed us to preserve some more of the context within our corpus. However, these methods took so long on a laptop, that we reverted to a simpler methods of encoding. If we do a project like this again we’d likely be better off offloading the computational tasks to the cloud, meaning we could employ more methods like deep learning and models like BERT.
There’s also a lot of experimenting left to do with the text preparation. We did not mess much with the words in the text during our preprocessing. Apart from lematising the words we removed only the most common English filler words. However, eliminating a more extensive list of words devoid of meaning could reduce some of the noise in our data and make it easier to identify the promises. We could also try training our model with text comprising n-grams, b-grams, or combinations that include parts of speech. All of this might provide more context for the machine learning model and improve accuracy.
We wrote several helper scripts intended to aid our manual review, such as by turning mentions of money into real numbers ($3m to 3,000,000). However, we only scratched the surface here. For instance, we didn’t delve much into language models and parts of speech to programmatically identify and remove re-announcements of election promises, which was all done manually. This might also have been achieved through its own machine learning model if we had trained one.
Bonus round: training an object recognition model to recognise novelty cheques and hardhats
While the methods above worked for the text of the Facebook posts, it couldn’t do much for the photos posted by politicians.
So, we found ourselves asking an important question. Could we use machine learning to spot photos of novelty cheques? Having another model in place to find big cheques and certificates in photos might pick up things we’d missed in the text, and also it was quite funny.
Giant cheques have made news in previous years – in 2019 when the former Liberal candidate for Mayo, Georgina Downer, presented a grant to a bowling club despite this practice usually being the domain of the sitting MP. A novelty cheque again made headlines in 2020, when Senator Pauline Hanson announced a $23m grant for Rockhampton stadium.
With this in mind, we trained an object recognition model to spot giant cheques. And from there it was a short step further to look at other common tropes of election campaign photo ops: hi-vis workwear and hardhats, cute dogs, and footballs.
We chose these as they were either already available in pre-trained models such as Coco, or had publicly available image datasets for model training.
For the object detection machine learning process we used the ImageAI Python library, which is based on TensorFlow.
ImageAI made it easy to get going without knowing too much about the underlying tech, but if we did it again I think we’d go directly to TensorFlow or PyTorch. When it came to figuring out problems with our models and model training there wasn’t much documentation for ImageAI, whereas TensorFlow and PyTorch are both widely-used with large communities of users.
Another option is an API-based approach, such as Google’s Vision AI, but cost was a factor for training models, which we’d need to do if we wanted to detect novelty cheques.
For each of the categories of hi-vis workwear, hardhats and novelty cheques we trained a custom YOLOv3 object detection model. Hi-vis was based on a publicly available dataset of 800 photos, while hardhat detection was based on a publicly available dataset of 1,500 photos. Dogs, sports balls and people were detected using a RetinaNet model pre-trained on the Coco dataset.
For cheques, we collated and labelled 310 images of giant cheques to train the cheque-detection model. Again, if we did it again we’d probably spend more time on the model training step, using larger datasets and experimenting with grayscale and other tweaks.
This time around, we trained the models on a PC (using the Windows Linux subsystem) with a decent graphics card and processor. While we also tried using Google CoLab for this, running the models locally was incredibly helpful for iterating and tweaking different settings.
Once we had the object detection models up and running, we collated Facebook photos for all Labor and Coalition MPs, candidates and senators for the 2019 and 2022 campaign periods, and then ran the object detection models over them. You can read more about the results here.
The biggest issue with our approach was the rate of false positives was quite high, even with the larger datasets used. That said, the approach was still much, much better than anything we could have achieved by doing it manually.
Note: while we would normally share the code for projects we blog about, the machine learning parts of this project involves a lot of data we’re not able to share publicly for various reasons. When we get the time we might add a stripped-down version of the project to GitHub later, and update here. You can however access the final election promises dataset here.
