Luke Briggs

The First Line of Code

Sat, 23 Jun 2018 16:42:40 +0000

The First Line of Code '/>

The First Line of Code

“The Guide says there is an art to flying”, said Ford, “or rather a knack. The knack lies in learning how to throw yourself at the ground and miss.”

-Douglas Adams

I always had a desire to look under the hood of the video games and software I spent so much time on growing up but I was greeted with a rainbow of semi-colons and syntax before immediately closing the lid of my laptop and retiring back to being merely an end-user.

It took until a computer science course at age 14 in my school before I was finally able to understand just the basics of what I could do with a keyboard. The language was python and the atmosphere wasn’t me alone in a bedroom at a monitor scratching my head in fury, it was a class effort. None of us understood what was going on and it made it all the more accessible.

In my current exploration of programming languages I see far too many people telling those learning to code of how you ‘must’ do this and I watch videos of monotonous commentators who half explain some concepts that fly over newcomers heads. I feel the steep learning curve that programming has is a mighty feat without an actual teacher or a series of quality YouTube videos that only come around once in a blue moon.

This blog will be a documentation of my journey from having basic grasps of a few programming languages through a higher education computer science course and my video-game creating antics on the side. It will detail my learning experiences and may help anyone else wanting to learn what the pathway is like.

A Game of Two Halves

Sat, 23 Jun 2018 17:20:33 +0000

A Game of Two Halves '/>

A Game of Two Halves

“It is a mistake to think you can solve any major problems just with potatoes.”

-Douglas Adams

The first videogame I ever played was Crash Bandicoot for the PlayStation One. I spent many an hour running my little orange friend across precarious ledges and away from Raiders of the Lost Ark levels of boulderage. As is usually the case with videogames during your early childhood, I imagine I didn’t get anywhere near as far as I think. I will never know though as not long into my tenure in videogames I suffered my first casualty. My PS1 and all its saves froze in a pixellated mess on my screen and in the words of John Cleese, “ceased to be”.

As I played more advanced and thought-provoking games I wondered what it would take to make my own and after numerous false starts I finally made my first game (sort of). The game , Fallen Stars, was made using [Construct 2] (https://www.scirra.com/). Construct is the piece of software that makes making games so easy you can create your first game within a day, rather than programming scripts and dealing with forces you utilise an incredibly powerful ‘drag and drop’ interface with the ability to create some pretty professional looking software. I recommend watching 3kliksphilip’s Game Making Journey for an almost videogame origin story as well an explanation of what these types of tools can do. I would thoroughly recommend those who don’t know where to begin to give these prototyping software ago as it is by far the way to get the quickest turnaround of a videogame from concept to executable.

Following my escapades in construct my game making took back seat as I switched to learning more programming languages past the rudimentary python skills gained at age 14 in school and final exams loomed in summer 2018. I furthered my knowledge of python and used websites such as Codeacademy to pickup the basics of JavaScript and C#; this is very useful in showing you what all different types of languages there are and the uses of them.

I would like to stress at this moment I only have a basic knowledge of these thee languages. Instantiation is as advanced as it gets at the moment, just wanted to reassure anyone wanting to learn to program that you don’t need to be an expert at all these to get a grasp of what programming is like or in order to get a good base for game development.

Fast-forward to now, summer 2018, and I have time for game development again and I wanted it to more closely link to my programming and software work. I chose Unity as my editor of choice for three reasons.

It allows me to use the C# skills I had already put the effort in learning.
It seems a lot more accessible than other engines as the structure seems more straightforward and you can get almost all features for free.
It is versatile. You can make both 2D and 3D games for pretty much all platforms.

It is with Unity that I went on to make my next game Ping.

Inspection & Dissection: Ping! – My First Game in Unity

Sat, 23 Jun 2018 17:52:33 +0000

Inspection & Dissection: Ping! – My First Game in Unity '/>

Inspection & Dissection: Ping! – My First Game in Unity

“Human beings, who are almost unique in having the ability to learn from the > experience of others, are also remarkable for their apparent disinclination to do so.”

Douglas Adams

Available at my project page

Why Such a Simple Game?

Because I wouldn’t have made a game otherwise. You need to affirm any skills in the most basic of games. Even if it kills you making a game so boring, at least you’ve made a game and proved that you can do it. It is an important lesson to learn that you will lose interest in a project and you will have to force yourself to carry on through the mid-development slog. To get around this you have to break a project into stages and set small goals. For instance, while making pong you may set yourself the goal of making a paddle that moves with keyboard inputs and can collide with things, next you might set the goal of making a ball that can bounce off of walls, next a working scoring system and so on. Another important thing to remember is you are almost definitely doing this as a hobby so don’t spend hours forcing yourself to do something you don’t want to do. Sometimes you do have to face it but it is almost always better leaving it and waiting until the next day to finish it when you have a clear head and can be bothered to continue. Burnout can often strike.

Where did you start?

First I only had the idea of making a game in unity, that’s it, so I watched a series by the excellent YouTube channel Brackeys that put me on the first step to knowing what I would have to do to make a game. I then decided on pong due to the reasons answered in the previous question. If I got stuck, I consulted the Incredible Unity Documentation, Google or follow a guide by Awesome Inc. Whilst the guide was necessary for some parts, specifically the ball bouncing, I made sure I did as much as I could with only the Unity documentation and Google searches as I often found on previous years of false starts I would just copy code and not learn anything whereas figuring out these things yourself means you are much more engaged and likely to remember it.

How do you think the game turned out?

As my first serious game, I am pleased. The game could be made more fun with more sporadic ball movements and a single player A.I. The paddle also glitches out slightly if you keep hitting into walls but it was meant to be nothing more than proving to myself that I can make a game in Unity. It has taught me the necessary skills and I have learned from the experience enough to move onto a slightly bigger project (not much bigger). Game development is a game in itself, you start off with a tutorial then make your way up increasingly difficult stages, you wouldn’t jump into a level 50 area when your only level 5 and you wouldn’t go straight to the glowing sea in fallout 4 when all you’ve got is a is vault jumpsuit and a baseball bat.

Where I Began

Wed, 27 Jun 2018 18:40:06 +0000

Where I Began '/>

Where I Began

“It can be very dangerous to see things from somebody else’s point of view without the proper training.”

Douglas Adams

As I have said before, my first outreach into programming began with a GCSE Computer Science course which was cancelled after 7 months for various reasons. Those 7 months were possibly the best way to get a strong foundation of how programming languages work and the general structure in which they follow. The lessons consisted of us being taught a feature (how to do ‘if’ statements, how to call functions, how to cycle through array, etc) and then being given a series of tasks that we had to create a program to complete. The first and simplest of these tasks was Bob’s Change:

"""Problem:
Bob would like to buy some sweets, which cost 50p.
Bob has some money but, unfortunately, Bob is only 3
and has no concept of money yet. Bob’s solution to this
is to just hand over the whole of his change and hope the
shopkeeper will sort it out for him.
The function bobs_change takes in a number representing
how much money Bob has handed over (in pence). It should print
the amount of change owed if he has enough, or
“Sorry Bob, you don’t have enough” if he is short."""

The program would then consist of if statements returning an answer after comparing the money Bob has to the price of a product. For those starting out programming, tests of these kind are a rewarding and fun way of practicing and reinforcing skills that you have learned. When learning a programming language I would strongly suggest using these sorts of tests after learning something new to ensure you both remember that skill better and to also give you a sense of progression as a new programming language can be hard to remember if it goes unused. The rest of the tasks we went through in that course can be downloaded here. And an entire assortment of more advanced tasks in many different languages can be found at codewars.com.

Really learning a skill and then practicing it is all you need to further your knowledge of programming. Once you have a grasp of basic features, you can then start to make more advanced programs using numerous packages and modules that give you the tools to create graphics and deeper features.

This was how I learned to program. It broke the stages up so I wasn’t daunted by the fact I only knew what for loops were and there was so much more in the language, I only focused on the task and I found with every task I got closer to an understanding where I could begin to realise what could be done and put me on good footing for future projects.

Making a Mobile App – Chapter 1: A Few False Starts

Sun, 09 Dec 2018 19:57:21 +0000

Making a Mobile App – Chapter 1: A Few False Starts '/>

Making a Mobile App – Chapter 1: A Few False Starts

“I may not have gone where I intended to go, but I think I have ended up where I needed to be.”

Douglas Adams

Self-learning is often like trying to solve a maze, you follow the first path you see as far as you can. You reach a dead end. Then use what you learned in that path to give you an idea of where you could go next.

I’ve been trying to find a new small project for a while now. I lacked the enthusiasm to spend time making games as I generally find it is something you need to take a running jump at with a fresh idea and be able to maintain that flair through the tidal wave of texture drawing, physics calculations and gameplay design. At some point I will go back to game development but I wanted something new.

It was around this time that the opportunity came up to develop an app that may or may not be used for a local business. It had the vague deadline that let me experiment but gave me the pressure to finish it, it let me design something that was professional and would have to be used practically and it was something I’d never done before on a platform I had never developed for.

Making a Mobile App – Chapter 2: A Bad Worker Blames His Tools

Sun, 09 Dec 2018 20:07:21 +0000

Making a Mobile App – Chapter 2: A Bad Worker Blames His Tools '/>

Making a Mobile App – Chapter 2: A Bad Worker Blames His Tools

“Don’t Panic.”

Douglas Adams

Finding the right tool for the job might be the most important thing I’ve learned >in my short time in the world of development. The classic conversation everyone who wants to get into development goes something like this:

New Guy: “What programming language is best.”

Pro: “There isn’t one.”

New Guy: “Ok. Well what programming language should I use for X.”

Pro: “It depends.”

New Guy: “Fine. Well what should I use if I want to use X for Y.”

Pro: “Whatever you are comfortable with.”

New Guy: “What do you mean comfortable. I’ve never programmed before, anyway its not as if a language is some sort of sofa. Hey! Where are you going! you can’t just leave me here!”

Google: “Hello.”

New Guy: “What.”

Google: “I’m your new best friend.”

There are so many use cases and things you might want to develop that almost every instance of someone’s development is bespoke to what they want. This is something that you have to have experiences before you understand it which makes it more useful for a new developer to learn while doing rather than learning then doing.

I had to find the right tools for what I wanted. My requirements for this app were based off of the time constraints I had, the lack of experience I had and the needs I felt the app had:

Needed to be easy to port to different platforms.
Had to be easy to integrate design and programming elements together.
Needed to be able to make a professional and modern look. 4.Didn’t require low level access

I settled on React Native with the Expo toolchain. I felt React was by far the best option I had. The only other options were using C# to make a Xamarin.Forms app which felt a bit like using a jackhammer to do teeth surgery or learning to use Java and Objective-C/Swift to do native development and use a completely different codebase, not only would I have to learn two languages and systems at once but also would need to fork out an initial cost of $99 for an apple developer account and develop on an Mac which I neither have nor want.

I’m using Expo because it means the app will be fully cross platform and I don’t need to interact with any of the awkward SDK’s and system specific stuff. Expo also provides a neat app that you run on your Android or iOS which connects to a local server with the app files and allows you to run your app within their app, no compiling, no apple developer account, no Mac. This made testing incredibly easy. Expo also handles keystoring for Android and all that fancy stuff.

Making a Mobile App – Chapter 3: The Setup

Mon, 10 Dec 2018 20:14:17 +0000

Making a Mobile App – Chapter 3: The Setup '/>

Making a Mobile App – Chapter 3: The Setup

“He attacked everything in life with a mix of extraordinary genius and naive >incompetence, and it was often difficult to tell which was which.”

Douglas Adams

Setting up your React/Expo app is fairly straightforward but does require using the command line and some basic knowledge. You can find further detail here but I’ll summarise it now.

Firstly you will need to download and install Node.js This is quite self explanatory as it has an actual wizard thus not requiring any command line interaction. Node is necessary for almost anything running local javascript. Next you’ll need to open the command line (found on windows by typing ‘cmd’ into the start menu) and run this command:

npm install expo-cli --global

You will then be asked to create an Expo account. Next you want no navigate to the directory where you want to store your app, this is done in windows by typing ‘cd’ followed by the directory. For example :

cd C:\Users\<Your Username>\Documents\app

Then run the commands:

expo init my-new-project
cd my-new-project
expo start

‘expo start’ will be the command you run in the directory whenever you want to run your Expo app. I would also recommend starting with the ‘tabs’ template as it places some very useful files and means you don’t need to set up your own navigation bars. After you have ran the start command your browser will open up with a QR code. Scan the QR code in the Expo app (Google Play and iOS) and behold your first expo app. If you have experience in ReactJS, you can dive into the files with your text editor of choice (I prefer VS Code) and follow the instructions both in the app and consulting the documentation. If you don’t have experience, I would recommend using the Codeacademy course on ReactJS and Javascript. Also remember Google is your best friend, if you don’t know something don’t be afraid to google it.

Making a Mobile App – Chapter 4: Fast and Flash

Mon, 10 Dec 2018 21:57:21 +0000

Making a Mobile App – Chapter 4: Fast and Flash '/>

Making a Mobile App – Chapter 4: Fast and Flash

“A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools.”

Douglas Adams

As this app will be for a business, it has to serve some purposes. First it has to present the company as a professional entity and also fit with its branding to provide not only a consistent experience but to also strengthen that companies place in your head, making them more likely to go to them rather than anyone else. Next it has to serve a point, it’s all well and good having some fancy spiel about company ethos and dedication to the craft but you want the user to have a reason to not only to install your app but to also keep using it. Finally, it has to be straightforward to use and foolproof. Well, nothing is foolproof proof. Well, nothing is foolproof.

Branding

The company colours were red, dark grey and white. It therefore makes sense for the app to only use the colours red dark grey and white, seems simple enough.

Eww, a red pastel background with black text and mismatch white buttons and icons. It looks like a tomato soup can just exploded in the microwave. It’s all about complimentary colours and keeping things clean. The main colour of any app or website shouldn’t be something outrageous or garish, you want to present professionalism and minimalism. It’s good to have an accent colour that pops and has your brand stand out but if you end up with too much it is both nauseating and ugly.

Ahh Much better. Character is nice but flamboyance is very easily annoying

Purpose

Like I say, your app needs a purpose otherwise what’s the point?

As you can see the app act doesn’t revolutionise e-commerce but what it does do is act as a hub for everything involved with this company. You can click a button and google maps opens up with the locations for shops and restaurants already plugged in and ready to plan a route. You can call to book a table for a restaurant and there are galleries of menus and images to see what products and dishes are on offer (As a side note if you are making a ReactJS app in Expo then react-native-image-view is a godsend, it is a bit fiddly to get working but it means you can keep using Expo without having to delve into Android files and java SDKs).

Foolproofness and Expandability

To make your app accessible you have to ensure it is something easy to navigate and intuitive to use. Since this is my first mobile app I felt it was more of a case of making sure it doesn’t look out of place compared to the modern selection of mobile apps that supermarkets and resteraunts have on offer. It uses clear icons and a navigation bar at the bottom which is the current style. I wanted the user to not be confused upon opening the app but feel comfortable in using it and having a smooth experience.

One of the brilliant things about React Native is the modules are not only easy to use but also offer a great deal of wiggle room to do what you want. For example, creating new screens is as easy as adding a function to that screens javascript add sticking a call to it in a main navigator script; this allows the room to expand quite quickly once you’ve got a baseline set giving scalability without what some projects have which is a feeling of trying to balance on one leg while trying to solve a rubik’s cube.

Making a Mobile App – Chapter 5: Go Your Own Way

Sat, 09 Mar 2019 20:47:46 +0000

Making a Mobile App – Chapter 5: Go Your Own Way '/>

Making a Mobile App – Chapter 5: Go Your Own Way

“Real programmers set the universal constants at the start such that the >universe evolves to contain the disk with the data they want .”

xkcd 378

I wanted to add a gallery for users to be able to scroll through a variety of images so I did a quick google search about how I was going to go about it and I found react-native-image-view. In 6 lines of code and an array of Imgur links you can have a decent and working image gallery, job sorted! Right?

In truth, it unsettled me. Was this cheating? Should I have created my own gallery? This package was better than anything I could come up with and something I wasn’t prepared to spend time on but I still felt like a part of the app was no longer mine.

These sorts of ideas are things I have seen keep cropping up across everything I have done. I see other people’s enemy movement packages in Unity and magic code on StackOverflow that does everything you need, but are these a step to far? Are you losing what it means to actuallly create something? I’ve had all these questions and I think I will always question myself about the balance between hard coding everything from scratch and using magic boxes that do the gruntwork for you, but for now I have decided it’s about scope. It is not within the scope of this project to make an image gallery, it is just a small feature in an app that is for doing something else. The amount of time it would take to create something like this is time that could be spent doing something else that is specific to this app and does require the time spent on it.

The way I intend to go about my projects is to learn what I need out of it and the crucial elements of it. If I’m making a game in Unity I would feel uncomfortable using someone else’s movement package, movement is such a core aspect of a game’s feel that I think I should have control over its nuance. Equally, I am ok with using an image viewer package as it means I can spend more time on bigger features and I will write my own image viewer if this one is no longer fit for purpose. This is a pet project and so it is really only my own motivation that is going to get it finished. The more time I spend on a time consuming feature, the less likely it is that the project will get finished. What I want out of this app is to come out the other side of it with something that is complete and to achieve this I need to pick and choose my battles.

Probably

Sat, 17 Aug 2019 20:49:25 +0000

Probably

We are currently in a scholarly purgatory. The time between July and September where you are attached to no institution, have no motivation and spend months fretting over how your past self will affect your future self (neither of which do anything for your current self)

At 11 am I received my AS results via e-mail (how modern) and received AAB + Distinction. I am happy with this. It’s literally the second-best results I could have achieved at this point, but there is still an unsettling feeling around me. Something is gnawing away at me that prevents me from enjoying this now.

During the summer months there isn’t a clear track to follow. No daily lessons, no scheduled assignments. Any events just seem to be smeared across what could possibly be discerned to be a date on close inspection. Nothing tangible appears to be happening in terms of progress for long term goals and as such there is no certainty.

The longer the time between terms, the more in flux my future appears to be. I could always fall back on the ‘everything is fine’ mantra but then it feels as if I’m not striving towards what I want, or that I’m setting myself up for future regret about not putting in the effort where it needed to be. My mind runs through permutations of events, “someone probably got 3 As, they’re the ones who are going to receive your university offer” I feel myself fretting. I shouldn’t fret, but I do. I know people who failed they’re exams and they have to repeat a year of their life, they would kill to be in my situation. There are also people who seem like demigods to me who always get perfect scores along with having elaborate and incredible side projects. To some my worries will seem preposterous, to others I will seem I don’t know how lucky I am. Perhaps I’m both these things and perhaps I’m neither.

But I’m not in their situation, I’m in my situation and can only act within the context of it. I want to achieve the best possible results I can, and I didn’t. I wasn’t far off; in fact, I was damn close but it doesn’t matter the distance. Out of the 216 combinations, I have satisfied 215, and the person inside me that wanted that last one remains dissatisfied.

Will 1 mark affect whether I get an offer from my preferred university? Probably not. Probably not. Probably is the most terrifying word in the English language. There are no certainties in life, everything can be prefixed with the word probably. Yet we feel the need to stick it before only a few things, I presume this is to deliberately remind us of the very large improbability of the task we about to undertake and then what is the point of even specifying a chance in the first place? Perhaps a wandering mind is a good sign. I’m not struggling to finance my education through a side job or agonizing over planning to resist a year, I have a supportive family and am grateful to my past self for not screwing everything up terrible.

So. Is everything fine?

Probably.

Inspection & Dissection: Dice-Jack

Tue, 24 Sep 2019 21:06:18 +0000

Inspection & Dissection: Dice-Jack '/>

Inspection & Dissection: Dice-Jack

As part of a course we had to make a game in 2 days that involved random dice and be made as a Windows forms application with Visual Basic. So we had to make a game in something that was the furthest you could get from a game engine with a language I had never used before. But hey, Rogue was made in 1980 and the graphics had to be ASCII so it could be worse.

The rest of the class generally stuck to the brief with 3 dice that you rolled and you got points for each dot.

As you can see, its rather simple. Theres 4 picture boxes, 4 labels, 2 buttons and a listbox. Dice are rolled, the dots are added up and the score is added to the listbox. After 5 rolls the game ends and the player is given their total score. But to know how I added further features we need to look at how this prototype was made.

Behind the prototype

Dim Die1, Die2, Die3 As Integer
Dim DicePoints As Integer
Dim random As Random
Dim RollCount As Integer = 0

So we have our variables declared in the convoluted VB way. Die1, Die2, and Die3 and the scores for the corresponding dice. DicePoints is the total of those 3 scores and rollcount keeps track of how many times the dice were rolled. Random is just an object we use to generate random numbers

Private Sub tmrTimer_Tick(sender As Object, e As EventArgs) Handles tmrTimer.Tick
        'Increments the load bar every tick
        random = New Random
        prbLoad.Increment(4)
        If prbLoad.Value = 100 Then
        ...
        Else
            'Sets dice to random number when the timer ticks before the load is full
            Die1 = random.Next(1, 7)
            Die2 = random.Next(1, 7)
            Die3 = random.Next(1, 7)

            MatchImage(Die1, pbxDie1)
            MatchImage(Die2, pbxDie2)
            MatchImage(Die3, pbxDie3)

        End If

On the board is a timer (inventfully called tmrTimer), it tick every 0.1s. Each tick it adds a bit onto the load bar at the bottom and randomizes the images on the dice. MatchImage is my own subroutine but all it does is take an integer and applies the correct face for it to whichever picture box I pass it.

Once the bar gets to 100, the fun begins.

If prbLoad.Value = 100 Then
            'Adds the necessary points on the random dice when the load bar 
            'reaches 100
            tmrTimer.Stop()
            prbLoad.Value = 0
            RollCount += 1
            DicePoints = Die1 + Die2 + Die3

We stop the timer ticking, we empty the load bar and we increment our roll counter before adding up all the spots on the dice, pretty simple.

'Adds Score to list
lstScore.Items.Add("Roll " & RollCount & ": " & DicePoints)
GameScore += DicePoints
DicePoints = 0

Almost done, we add our points to the listbox with some formatting. Add the points to the total score for the game and then set it back to 0.

If RollCount >= 5 Then
    MsgBox("Your game score was " & GameScore,, "Game Over")
    tmrTimer.Stop()
    btnStart.Enabled = False
    btnReset.Enabled = True

End If

Then we check whether they have reached the magic 5 rolls, if so we tell them their score and let them reset the game

There is also some initialisation code that assigns images and there are event handlers that make the buttons work but this is the main logic. If you would like to see how that stuff works, you can find it at the github repo

The Evolution

The following won’t include too many code snippets since it is slightly more extensive (the rushed deadline also means it resembles a tin of Heinz spaghetti in a tumble dryer but we’ll keep that between us)

Lets break this down.

First we have a background that was shamelessly made in powerpoint.

The rules, scores, and title are all labels. The bar on the right is made up of the same button and listbox items as before but with its properties edited to change colour, font and border style.

The Rules have been changed to the following:

The goal is to get as close to 21 points without going over
There is a Player and a Dealer
Each contestant has 2 dice
The player goes first
Each dice spot is worth 2 points
If either the player or dealer goes over 21, they lose the game
After the first roll, if the player has not gone bust, they can choose to ‘stand’ or ‘hit’
If they ‘hit’, the dice are rolled again and the player can choose to add the points of either die or the total of both dice
The player must add a score after hitting, even if it means they will go ‘bust’
In the event of a tie, the game goes to the dealer

The rules are based on those by James Yates

Instead of a big chunk of code I’m going to show you the process of a turn through our good old friend the flow chart. They can be intimidating but I’ve tried to make it in a way that is easy to follow and you should be ale to get a grasp of what the code is doing behind the scenes.

There you go, who needs a hundred lines of bolognese when you can have pastel shapes?

Those who played the game may also notice this is the first time I’ve included a computer-controlled opponent, ‘The Dealer’. The dealer is like the player but has the advantages of being able to go second and win in the result of a draw. The dealer’s choice code is as follows:

If DealerCanChoose1 And Not (DealerCanChoose2 Or DealerCanChooseBoth) Then
    DealerScore += Die1 * 2

ElseIf DealerCanChoose2 And Not (DealerCanChoose1 Or DealerCanChooseBoth) Then
    DealerScore += Die2 * 2

ElseIf DealerCanChooseBoth Then
    DealerScore += (Die1 + Die2) * 2

ElseIf DealerCanChoose1 And DealerCanChoose2 Then
    If Die1 > Die2 Then
        DealerScore += Die1 * 2
    Else
        DealerScore += Die2 * 2
    End If
End If

If Not (DealerCanChoose1 Or DealerCanChoose2 Or DealerCanChooseBoth) Then
    'Dealer goes bust if no options are available
    If Die1 < Die2 Then
        DealerScore += Die1 * 2
    Else
        DealerScore += Die2 * 2
    End If
    DealerStand()
    tmrDealer.Stop()

ElseIf DealerScore > GameScore Then
    'Dealer stands if their score is more than the player's
    DealerStand()
Else
    DealerRoll()
End If

Before this code I run checks that see which options are available after a dealer rolls (these are the DealerCanChoose variables). Here there is a series of conditionals that look at the options and picks the best one for the dealer.

If there is only one option available, then the dealer will pick that one. If there is more than one option available then the dealer will pick whichever will get them the highest score that turn. If there are no options available then the dealer is bust and they will stand. The dealer will also stand if they have a score higher than that of the player. The dealer will continue to roll as long as they are lower than the player and their current score is under 17.

And thats the game! The quickest project I’ve ever done and although its rough around the edges the gameplay is solid and I got to grips with windows form’s implementation of an event-driven program. You can find the full source code here

The Layperson’s Guide to LaTeX

Wed, 06 Jan 2021 22:06:22 +0000

The Layperson’s Guide to LaTeX '/>

The Layperson’s Guide to LaTeX

As part of my degree I will have to write many words into many documents and submit them all as part of assignments.

There was a time when we were young, stupid and thought that the peak of document formatting was WordArt, a drop shadow, and rainbow II. We would stick borders on our .pub files and feel superior if our PowerPoints had a dissolve transition. As we age past primary school we begin to conform to the world’s sensibilities that Comic Sans is not an adequate typeface and having every colour on the spectrum is a way to actually guarantee some of your text will be unreadable.

Since we are forced to conform, we might as well do it to perfection, and you can only go so far in a tool that still thinks that word processors need a background ‘water droplet’ texture fill.

What is this LaTeX and Why is it Not Pronounced Like That?

If you want a tool that focuses on the nicest looking documents without caving into the design requirements of 7 year olds, you have to go back to a time before texture fills. Actually you have to go back to a time before computers could even show pictures. Nice looking documents (i.e had legible fonts) had to be obtained using specialised typesetters that cost as much as a house, either that or you’d use a literal printing press – this was the time in which TeX was created. LaTeX is built on TeX and is its more approachable step-son. LaTeX uses a mark-up language to design its document with the idea being that you focus on the content while the engine works out the best formatting.

Isn’t This a Bit of a Faff?

It depends on your personality. If you are happy with documents that are only adequate, then continue in your mediocrity. But for those of us who seek perfection and see neuroplasticity as a fundamental attribute, LaTeX offers a sterling reward for your efforts.

How Good Does it Look?

In the days when people weren’t made of pixels, I spent the time making my own word template. It had wonderful serif headings, all the styles used Word’s tools so it did as much of the heavy lifting as possible, and all the font sizes were made just right.

The results from Word are as follows:

It looks okay. It looks far better than what some people create in Word. Even creating something okay looking feels like a hack though. Having nice paragraph spacing underneath headings required me to individually change all the line spacing options by hand. Anyone who has ever tried to implement code into a word document also knows that it will require you to sacrifice your firstborn.

Now, for a LaTeX document:

No time spent messing with templates, no changing font sizes, and it probably took me less time to make a document that looks even better. The best thing about LaTeX is that because everything is done programmatically, it can have an integration that is mind boggling. For instance, if you want to insert a segment of a python script in your document you dont actually need to copy and paste bits into your .tex file. You can just tell it where the script is, give it the line numbers and it will display and format it all for you.

Where Do I Begin?

This is for those of a Windows disposition

First install a distribution of LaTeX called MiKTeX. MiKTeX has everything you need and will make the whole experience as easy as possible.

Our code editor is called TeXworks, so open that up and lets write some LaTeX.

LaTeX follows a syntax of \command[option]{parameter} and we start a document off by selecting the type of document we want to create, and then starting and ending said document. There are a number of document classes but article is the one recommended for most documents.

% Start of Document (Comments are denoted by a percentage at the start of the line)
\documentclass{article}
\begin{document}
I'm so pretty
\end{document}

And we get:

Currently it looks like a note left by a serial killer with access to a typewriter so lets stick our name on it.

The title, date, and author of your document all have dedicated tags in LaTeX. They are \title \date and \author. We then tell LaTeX to show all these on the screen by using \maketitle

% Start of Document (Comments are denoted by a percentage at the start of the line)
\documentclass{article}

\begin{document}

\title{My Pretty Document}
\author{J. Smith}
\date{23 November 1963}
\maketitle

\end{document}

Output:

You can hear groups of your preferred gender(s) running to throw themselves at you as we speak.

Now for some content. Once again there are various different types of sectioning you can use but we will use the pretty straight forward \section

% Start of Document (Comments are denoted by a percentage at the start of the line)
\documentclass{article}
\begin{document}

\title{My Pretty Document}
\author{J. Smith}
\date{23 November 1963}

\maketitle

\section{This is a subtitle}
I can just write text under the section tag and it will all be made nice

\section{This is my second subtitle}
I just add another section tag and it will become the second section in the document. Pretty cool!

\end{document}

Output:

Positive gasps emanate from the crowd at the sight of such serifs. You will notice that LaTex automatically added heading numbers and changed the weight, face, and size of the type to better suit a heading. There are all sorts of things you can do that I haven’t covered like how it can do bibliographies for you and organise images and figures correctly. There are also things that are beyond my knowledge like drawing your own 3d graphics within source. All of it can be taken one step at a time. If you don’t know something, look it up, learn how to do it, and implement it – it’s all pretty approachable.

LaTeX isn’t perfect in its implementation, only its possible output. For instance, the answer to the question ‘how do I insert an svg’ is ‘use a pdf instead’ but I hope I have given you a taste of a better alternative to Word, even if it does introduce the possibility that your reports can have syntax errors.

Further Notes

If you are writing your reports in LaTeX like me then you will need the correct bibliography formatting. LaTeX offers a few styles by default but I have learned there is no such thing as a standard and my University (Newcastle) uses its own guidelines. The process for creating your own style involves both a command line tool and further tweaking using a bespoke programming language using REVERSE POLISH NOTATION, which is what happens if Christopher Nolan got his hands on a compiler. To save my fellow students time, I created a style for Newcastle’s bibliography style and it is hosted on GitHub.

Inspection & Dissection: This Site!

Wed, 03 Feb 2021 16:39:25 +0000

Inspection & Dissection: This Site! '/>

Inspection & Dissection: This Site!

Look at me with a fancy website. We're about to get meta as we discuss how this site you're exploring right now came to be.

Why have a website?

I want something to point to when someone asks 'so what have you done?'. I also need it for when I ask myself the same thing. Sometimes I wonder how much I have actually done in my spare time, and a site like this helps me to remember that I haven't wasted all of my spare time on Minecraft.

Why make your own website?

Here in Computer Science land: men are real men, women are real women, and small furry creatures from Alpha Centauri are real small furry creatures from Alpha Centauri; thanks to these huge revelations in the field of existentialism we take it upon ourselves to walk the road already taken.

The real reason is I ought to have side-projects, more than I used to. I have more time, not just because of COVID but also because Computer Science is now the only academic subject I have to pay attention to, whereas most of my time in Sixth Form was spent on Maths and Further Maths. It made reasonable sense for the first project to be a platform for my other projects and this is that platform.

Where did you start?

I knew I wanted to make a website with a Flask back end for 2 reasons:

I wanted a back-end that could enable things like logging in and writing blog posts all within the site.
I vehemently hate JavaScript and would prefer to do as much with python as possible. In fact this whole site doesn't have any JavaScript. You can inspect element and see that everything front end is done through CSS and HTML; this is certain to change but I will hold out for as long as I can.

For the uninitiated, websites have 2 parts: A front end and a back end. In a simple form, the front end is everything you see (HTML, CSS, JS) and the backend is everything you don't (Databases, Form requests, etc). Flask is a python framework that allows the said language to be used to serve all your requests.

I had used Flask in the past whilst following the Harvard CS50x course back in 2019 to make a fake stocks app (💎👐). This time round, to re-familiarise myself I followed this excellent tutorial.

After you are let loose on your own (especially in web dev) I found you have to have goals and a purpose to what you are doing otherwise it'll never happened. I persevered through the trial-and-error process that is CSS

I used a couple of sites for reference when I was making the design for the sight. These were the sights of: my new friend George Wood, the blog of Casey Liss, the blog of Marco Arment, and Hypercritical (No prizes for guessing which podcast I listen to).

What extra things have you done?

I'm honoured at your presumption that I go above and beyond in the call of duty. The main thing I have done is make all my posts in markdown. Yes, all my posts are entirely written in markdown which makes me able to write nicely structured documents quickly. The raw text of the markdown is stored as a regular text field in the sqlite database but at the stage when the backend grabs the field from the database, jinja parses it as html using the python module mistune. To spice the formatting up even further, since I will be probably be using code snippets, I added the syntax highlighter pygments. In fact here is the code for the highlight renderer being rendered in it:

import mistune
from pygments import highlight
from pygments.lexers import get_lexer_by_name
from pygments.formatters import html


class HighlightRenderer(mistune.Renderer):
    def block_code(self, code, lang=None):
        if lang:
            lexer = get_lexer_by_name(lang, stripall=True)
            formatter = html.HtmlFormatter(style='monokai', noclasses=True)
            return highlight(code, lexer, formatter)
        return '<pre><code>' + mistune.escape(code) + '</code></pre>'

It goes without saying that this website will continue to be updated as I change my mind on a few things and wish to add more functionality. On my list of todos are:

A switch between light mode and dark mode
A way for people to follow the blog (maybe through RSS)
A way to better list out blog posts (currently all are listed on one page which will probably become cumbersome when they grow in number)

In the meantime I hope you come back and see if I can offer you any informative content.

Goodbye Windows, I Hardly GNU ya

Sun, 07 Feb 2021 21:12:30 +0000

Goodbye Windows, I Hardly GNU ya '/>

Goodbye Windows, I Hardly GNU ya

Preface: I should state that this is not an evangelical sermon, Linux isn't for everyone but it is for me. This is merely a detailing of why I have chosen to switch Operating System

I have, over the course of the past year, consumed pretty much every interpretation and adaptation of Sherlock Holmes out there, from the original works to tangentially linked shows such as the excellent House MD; one thing I have come to understand is that you can only deduce correctly if your initial assumptions are correct. Logic is useless if its underlying axioms are unstable.

So we turn to my current assumption when I use computers "Everything must be done in Windows". I have used Windows all my life, all the software I need is on Windows, and all other operating systems are either anaemic in their UX and feature set or are incapable of running on my hardware. It just stands to reason, or does it?

Lets look at that first point, 'I have used Windows all my life'. That is not a very good reason, and the weakest of all. It is indeed true but it shouldn't have any bearing on choice of Operating System, it is an unquestioned ideology rather than any valid argument.

The second axiom, 'all the software I need is on windows', was probably true for quite a long time. For instance, if I wanted to play any windows games I would have been bang out of luck in days gone by, but tools such as Lutris and the strides made in WINE have made that side of things perfectly capable for my needs. Needless to say Mac is straying further away at this point with their move to ARM so the assumption still holds true for them. Outside of gaming, all the apps I need are actually available on Linux. My academic work in Computer Science lends itself perfectly to Linux. All the IDEs I use (the JetBrains suite) are available and the terminal is actually more conducive to my studies compared to the limited Unix-like aliases in the Windows Powershell. The only notable exception on Linux is the lack of the Microsoft Office Suite, but I was quite staggered when I realised that I don't actually have any need for it. My long form writing is now done in LaTeX (see my reasons for that here), and I don't use Excel or PowerPoint. Should the need ever arise, there is the native LibreOffice suite, and the cross platform Google Docs. And if compatibility is a necessity then Microsoft also now offers a web app version of the suite.

The second point links to the third, that other operating systems have terrible UX. The king of UX is obviously MacOS, but that obviously doesn't support my hardware or the ability to run the occasional windows game. I had always perceived Linux as having terrible UX because I always just viewed through the lens of someone who has used Windows their whole life. I always just assumed (once again question your assumptions) that how a Linux distro shipped was how it was. I thought I would have to be stuck with the bland, Ubuntu default GNOME desktop and icons with their washed out colours and pre-iOS 6 style realism. I was completely wrong about this and now we are going to look at where Linux, my new Operating System, is actually better for my needs.

The Linux way of thinking

Linux is about what Computers were built to do, and it is about the things that us Computer people love. GNU/Linux is a whole software ecosystem that is far greater than what Windows could ever hope for and only rivaled by what Apple has cultivated on its platforms. Unlike Apple, however, GNU/Linux also offers the user complete freedom in every aspect, and there are levels to this. Yes there are demigods that are compiling their own kernels but because there is such a community driven approach, other people have performed every possible layer of abstraction. Over the decades other people have been putting rungs at all levels of the ladder for other people to step on. Because Linux is built for the community by the community, it doesn't fight me.

If I wanted to change the interface on windows, I would need to install several third party apps, each using their own design language and frameworks all while eating up huge amounts of RAM as they do so. In Linux its as easy as choosing an overall desktop environment you like and customisation is welcomed with open arms as a basic human right. Surprisingly, Linux also has an incredibly consistent design language because of how its built. In Windows you have all sorts of frameworks from many decades that all conflict. You'll go from a UWP app with its tile based design ideas, and then open a WPF app that has a reasonable flat aesthetic and some okay ideas, before finally finding one of your applications uses WinForms with all of its tabs and ugly green loading bars. Almost everything on Linux using either the Gtk or Qt GUI libraries, and because of this everything looks consistent. Most Desktop Environments also let you either make or download your own themes for these graphics libraries. As a result you can make all the colours, buttons and menus in the ENTIRE Linux ecosystem look how you want.

The Linux way is for people who care about these things. If you care about things like the same apps using the same frameworks, or the fact that you can use package managers to allow apps to auto-update and avoid redundant dependencies, or how you have a whole community of people who's only incentive is to make good software.

Technical Details

Distribution: Manjaro

Desktop Environment: KDE Plasma

Theme: Breeze Dark

Icon Set: Reversal

Background: (Imgur) (Mirror)

Why Manjaro?

I chose Manjaro, at first by design and then retroactively I realised that I would have had to go with something like it anyway. I wanted a distribution that didn't come with the usual GNOME look, something that either looked excellent at first glance or had extensive customisation capabilities, and something that was pretty light on what it came with. I ended up narrowing it down to Manjaro and Elementary OS. In the end I chose Manjaro between the two because Elementary seemed slightly far behind in its update cycle compared to the Ubuntu it is based off and its never fun to start off with something knowing there is a major version incredibly close. It turned out in the end that Manjaro was really my best choice between the two anyway thanks to its easy switching between Kernels. I currently need to be running the latest experimental kernel for my touch pad to work.

You Can Now Follow Me on RSS!

Thu, 18 Feb 2021 20:52:00 +0000

You Can Now Follow Me on RSS!

I had a bit of time today and I wanted to do something so here we are. Thanks to the incredible FeedGen python library, you can now point your RSS reader to here and get updates on when I write a new post.

Inspection and Dissection: Type or Die

Fri, 05 Mar 2021 12:11:40 +0000

Inspection and Dissection: Type or Die '/>

Inspection and Dissection: Type or Die

Itch.io Link

For the past 5 days I have done what I have failed to do many times before, complete the process of making a game.

The game was for the NUCATS Game Jam. NUCATS is the Newcastle University Computing and Technology Society. It was made in Unity (with great help from Brackeys), and it was really the first reasonably sized game I've made. The only other thing I've ever made in Unity was Ping which, as the name suggests, was a pong clone.

Type or Die is a top-down zombie shooter where it is not about how good your aim is, but how fast you can type. Waves of zombies come at you in all directions, with the direction they come from determining what key to press to shoot at them (e.g a zombie coming from the top right would be shot at using the 'p' key).

The game jam has come in a time between some pretty busy weeks so I have surprised myself by actually getting a finished game out. I even contemplated not taking part at all after the first day because I felt I just wouldn't be able to do it. I started off (as optimistic idiots often do) full of goals with zero plans. Every step of the way I was decreasing scope to match my skill and enthusiasm, and its where those paths crossed that the game was made. I knew that the most complex thing I could make in the time with limited skill was a 2d, single-screen game. Guns are an easy way to make the mundane exciting, so the game became a shooter, and wanting to have the computer do all the work for you meant that it became a wave shooter for replayability.

Every waking moment from Monday to Thursday was spent either watching Suits or making this game, and the sheer intensity of such an endeavour in the middle of some pretty hectic University team projects made me realise one thing, I will probably not make another game for a very long time. If you have a gander at my projects page you will see that all of them are games, so you may think that I really enjoy game-making. But in retrospect I think I only made games as an outlet for programming rather than an outlet for artistic ideas. Creation is a painful process, so you have to see the purpose in it. As much as I am proud of what I achieved with each of the games I made, I can't see them as providing a service.

My perspective on games has completely changed in the past year, I place a huge value on the incredible narrative experiences offered by the likes of 'God of War' or 'The Beginners Guide', these are games made by people with something to say and the skill to say it. There is certainly a place for arcade shooters, but if I don't feel passionate about the things I create then I will never see them as worth the time I put into them.

Type or Die marks the end of the 4-year-long adventure that started this whole blog off, I suppose it is fitting that it should end with another top down arcade shooter. The game is pretty fun, the zombie death animations are my favourite part (also an original theme made by my musically inept hand).

More projects lie on the horizon, and the times they are a changin'.

Where Hugo I Go

Mon, 03 May 2021 20:06:07 +0000

Where Hugo I Go '/>

Where Hugo I Go

As you are probably aware, I have redecorated. I said in a previous post that my reason for making my own site in the first place was as a central hub to point people to all the things that I do. Despite the fact that from the outside it looks like I haven't done a whole lot since, behind the scenes I have my biggest project yet in development (check out my GitHub if you want a sneak preview of that). It was while working on my project that I realised that I shouldn't have a site that requires my to ssh into a VPS and pull a git repository every time I want to add a post.

Technology

Behind the scenes, the site is hosted on GitHub Pages. The site is generated using a static site generator called Hugo with a slightly tweaked version of the LoveIt.

With a static site I can remove server costs completely, have a better looking site, and make it easier to add new posts.

Because each post is now a document rather than a database field, you can also stick index.md at the end of any post to get the raw markdown for that post. You are also able to view the source for the site at my GitHub

New Logo

Along with the revamped site, I have created a new logo to go with it; it is probably the best looking thing I have ever drawn. The logo takes inspiration from the IBM PCJr with a few more rounded corners, thinner bezels, and a more minimalist feel.

It was made in Inkscape with the basic shapes and the font is Roboto Mono.

Flatpak: Instructions Not Included

Sat, 22 May 2021 19:08:56 +0000

Flatpak: Instructions Not Included '/>

Flatpak: Instructions Not Included

There is a very peculiar thing I have found in software development, it is incredibly difficult to actually get software on other people's machines. Even compiled languages can have difficulty.

Context

I am elbow deep in a very large personal project at the moment (all will be revealed soon) that has so far taken a month. The project is getting to a stage now where deployment methods have to be evaluated. From the beginning I wanted the project to be fully cross platform across Windows, Linux, Mac with a singular codebase. Because of these specific requirements, I went with the Qt GUI framework, it is cross platform, runs natively on each platform, and has bindings for Python (my preferred language).

Because nothing is ever easy, there are actually two different python bindings for Qt. There is PySide which was created by the Qt Company themselves. It has a less restrictive license, but seems to have less of a community around it due to it being the newer of the two bindings. The other binding is called PyQt and came first, even being called python3-qt in the fedora package manager. When it comes down to coding, there is pretty much no difference between them since they are all just pythonic fronts for the same C++ back-end. For the whole project up until recently I was using PySide since it is the same in every way except for a better license.

Python Deployment

Python, as it is shipped as CPython, does not (by default) compile to machine code. You'll notice a parenthetical and a subordinate clause in the previous sentence, that's how windy this road goes.

There are systems that will turn your python code into binary executables, I was using PyInstaller to get the job done for me. I'd run it on Linux, Windows, MacOS and all my code would be packaged in to binary form; there was the issue of non-python dependencies I had but I could find ways around that. So there I was with all my binaries in a row and feeling confident about how I'm going to deploy my application.

Linux Binary Woes

My current main machine runs Fedora Linux, this is a distribution that is kept reasonably up to date and, at the time of writing, has a new desktop environment and newer kernel than other non-Arch distros. All of this meant that when I tried to run my Linux compiled program on Ubuntu (A major distro I want to support), I get thrown an error that Ubuntu doesn't have the correct version of glibc (the dynamically linked C libraries). Sticking with the standard glibc versions that come with each distro, GNU/Linux is backwards compatible but not forwards compatible. That is to say that a program compiled on an older distro will run on a newer distro but not vice-versa. The solution to this seems straightforward doesn't it? Compile the program on an older version. So I loaded up an Ubuntu LTS VM, compile the program, works like a charm. I try to run that same executable on my main Fedora machine and I get an error message:

Settings schema 'org.gnome.settings-daemon.plugins.xsettings'
does not contain a key named 'antialiasing'

The program does not run. This is without doubt a bug in Gnome 40. But it is a bug that happens 100% of the time, it may get fixed, it may not. My application is either incompatible with anything other than my own machine, or compatible with everything except my own machine. Of course I could have two separate binaries, but that would be a bit of a hassle to maintain. In comes what I thought would be my saviour, Flatpak.

Build Systems

Linux has 3 main methods of deploying applications. Shipping binaries, making per distribution packages, and containers (of which flatpak is one). Binaries are, as we've seen, version dependent, packages are distribution dependent, and containers run across all machines but have extra bulk since they package there own dependencies rather than relying on system libraries.

Throughout my exploration of deployment options, I have made all three. I have documented my results of binaries, and I also went on to make an RPM package (the packaging system that Fedora uses).

Name:           Pepys
Version:        0.3.0
Release:        8%{?dist}
Summary:        A straightforward markdown journal

License:        GPLv3
URL:            https://www.lukebriggs.dev/pepys
Source0:        pepys.tar.gz

Requires:       python3, python3-wheel, python3-pip, pandoc, enchant, wkhtmltopdf, python3-qt5, python3-qt5-webengine, python3-regex, python3-num2words, python3-pypandoc, python3-enchant, python3-setproctitle, texlive-mdwtools
BuildRequires:  python3-pip
%description

%build
MAIN_DIR=%{buildroot}/usr/share/pepys
APP_DIR=%{buildroot}/usr/local/share/applications
mkdir -p ${MAIN_DIR}/temp
mkdir -p ${APP_DIR}

tar -xzvf %{SOURCE0} -C ${MAIN_DIR}/temp
if test -f ${MAIN_DIR}/temp/src/resources/base/config.json; then
rm ${MAIN_DIR}/temp/src/resources/base/config.json
fi
if test -f ${MAIN_DIR}/temp/src/resources/base/wordlist.txt; then
rm ${MAIN_DIR}/temp/src/resources/base/wordlist.txt;
fi

cp -r ${MAIN_DIR}/temp/* ${MAIN_DIR}
rm -rf ${MAIN_DIR}/temp


printf "[Desktop Entry]\n \
Type=Application\n \
Name=Pepys\n \
Categories=Office;\n \
X-GNOME-FullName=Pepys\n \
Comment=A straightforward markdown journal\n \
Icon=%s/src/main/resources/base/icons/appicons/icon.svg\n \
NoDisplay=false\n \
Exec= python3 %s/src/main/python/main.py\n \
Path=\n \
Terminal=false\n \
X-GNOME-UsesNotifications=false\n \
StartupWMClass=Pepys" "/usr/share/pepys" "/usr/share/pepys" > $APP_DIR/Pepys.desktop
pip3 install PyPDF4


%files
/*

%post
chmod -R 777 /usr/share/pepys

#%license add-license-file-here
#%doc add-docs-here



%changelog
* Sat May 15 2021 Luke Briggs <lukebriggs02@gmail.com>
-

NOTICE: The above is a script used for testing, it is mostly functional but don't use it as a guide for how to do things properly

The above is a script you feed into rpmbuild and it will generate a .rpm file which Fedora users can use to install your software. You can see that all it really is is some metadata listing app information, dependencies (these dependencies are ones that are inside the fedora packaging system), and a shell script to say what gets installed and how. Because all the dependencies are listed, and are all from the the distros package manager, the file size of an rpm only consists of the files that you supply. In my case it was some python source files and some vector icons, totalling 1.2MB. Of course the full size of the app still needs to be downloaded but the other 150MB or so come in the form of system libraries that other apps can also use. This is the classic Linux method of software deployment, it is lean, and ensures compatibility with a particular distro. The problem comes when looking at other distributions. Some distributions also use RPM (CentOS, RHEL, OpenSUSE) but have different packages, and so need special cases in the file. Other distributions use a different package management system, Debian and derivatives use .deb files for instance.

Flatpak

Flatpak and it's rival Snap aim to solve the dependency hell by shipping everything together in one big package. You avoid incompatibility at the cost of redundancy. The cover image of this post is my flatpak spec file that gets fed into flatpak-builder. And you may notice that it works in a similar way to a distribution's own package manager.

We have metadata along with base dependencies

app-id: dev.lukebriggs.pepys
runtime: org.kde.Platform
runtime-version: '5.15'
sdk: org.kde.Sdk
add-extensions:
  org.freedesktop.Sdk.Extension.texlive:
    directory: texlive
    subdirectories: true
    autodelete: true
    version: '20.08'
command: runner.sh

Because everything is in a sandbox, we have to declare what parts of the host system we have to use

finish-args:
  - --share=ipc
  - --socket=x11
  - --socket=wayland
  - --socket=pulseaudio
  - --device=dri
  - --filesystem=home
  - --share=network
  - --env=PATH=/usr/bin:/app/bin:/app/texlive/bin:/app/texlive/bin/x86_64-linux:/app/texlive/bin/aarch64-linux

And then comes the rest of our dependencies which have to be files we supply, along with the build script we use to actually do what we want with the files.

The Trouble with Python, Qt and Flatpak

All was going swimmingly until I had to get my Qt app in flatpak. 2 of my app dependencies are installed by effectively copying the prebuild binaries into the flatpak sandbox.

- name: pandoc
    buildsystem: simple
    build-commands:
      - ls
      - install -D bin/pandoc /app/bin/pandoc
    sources:
      - type: archive
        url: https://github.com/jgm/pandoc/releases/download/2.13/pandoc-2.13-linux-amd64.tar.gz
        sha256: 7404aa88a6eb9fbb99d9803b80170a3a546f51959230cc529c66a2ce6b950d4c

  - name: wkhtmltopdf
    buildsystem: simple
    build-commands:
      - cp -r local/* /app/
      - ls lib
    sources:
      - type: archive
        url: https://github.com/wkhtmltopdf/packaging/releases/download/0.12.6-1/wkhtmltox-0.12.6-1.centos8.x86_64.rpm
        sha256: 5cc267d54fe3f144729f31432a165e028b689583c53cfee0a01d52644ab280d9

One of my dependencies is actually built and compiled within the flatpak container. (Given no buildsystem option on archives with makefiles in them, flatpak will automatically compile and install them)

  - name: enchant
    sources:
      - type: archive
        url: https://github.com/AbiWord/enchant/releases/download/v2.2.15/enchant-2.2.15.tar.gz
        sha256: 3b0f2215578115f28e2a6aa549b35128600394304bd79d6f28b0d3b3d6f46c03

So we are back to the dilemma of using source files or prebuilt binaries, you'll see later that the solution is a mix of both.

One option is just shipping a compiled version of my program in a flatpak. The issue is that flatpak's glibc is once again older than my main machine so all flatpak builds would have to be done in an Ubuntu VM, not ideal. Next up comes building from source in the flatpak. My plan for the process was this:

Get python and resource files in the flatpak
pip install all dependencies

Sounds simple enough, but flatpak didn't like me communicating with the internet while installing (I tried share=network on individual modules and it didn't work) So, each python library had to be down as it's own python module with a link to the tar balls on pypi. Flatpak has a tool for such an occasion called flatpak-pip-generator so the line in the flatpak yml

- python3-requirements.json

is a pointer to a file actually containing all pip dependencies

{
    "name": "python3-requirements",
    "buildsystem": "simple",
    "build-commands": [],
    "modules": [
        {
            "name": "python3-regex",
            "buildsystem": "simple",
            "build-commands": [
                "pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"regex\" --no-build-isolation"
            ],
            "sources": [
                {
                    "type": "file",
                    "url": "https://files.pythonhosted.org/packages/38/3f/4c42a98c9ad7d08c16e7d23b2194a0e4f3b2914662da8bc88986e4e6de1f/regex-2021.4.4.tar.gz",
                    "sha256": "52ba3d3f9b942c49d7e4bc105bb28551c44065f139a65062ab7912bef10c9afb"
                }
            ]
        },
        {
            "name": "python3-num2words",
            "buildsystem": "simple",
            "build-commands": [
                "pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"num2words\" --no-build-isolation"
            ],
            "sources": [
                {
                    "type": "file",
                    "url": "https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz",
                    "sha256": "49b3a825280bd66b3aa83585ef59c4a8c82f2c8a522dbe754a8bc8d08c85c491"
                },
                {
                    "type": "file",
                    "url": "https://files.pythonhosted.org/packages/33/db/76f1151a1b0cfad532d41021b77cd231495bf72c47618166f92dcdff2ebe/num2words-0.5.10.tar.gz",
                    "sha256": "37cd4f60678f7e1045cdc3adf6acf93c8b41bf732da860f97d301f04e611cc57"
                }
            ]
        },
        {
            "name": "python3-pypandoc",
            "buildsystem": "simple",
            "build-commands": [
                "pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"wheel\" --no-build-isolation",
                "pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"pypandoc\" --no-build-isolation"
            ],
            "sources": [
                {
                    "type": "file",
                    "url": "https://files.pythonhosted.org/packages/ed/46/e298a50dde405e1c202e316fa6a3015ff9288423661d7ea5e8f22f589071/wheel-0.36.2.tar.gz",
                    "sha256": "e11eefd162658ea59a60a0f6c7d493a7190ea4b9a85e335b33489d9f17e0245e"
                },

                {
                    "type": "file",
                    "url": "https://files.pythonhosted.org/packages/d6/b7/5050dc1769c8a93d3ec7c4bd55be161991c94b8b235f88bf7c764449e708/pypandoc-1.5.tar.gz",
                    "sha256": "14a49977ab1fbc9b14ef3087dcb101f336851837fca55ca79cf33846cc4976ff"
                }
            ]
        },
        {
            "name": "python3-pyenchant",
            "buildsystem": "simple",
            "build-commands": [
                "pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"pyenchant\" --no-build-isolation"
            ],
            "sources": [
                {
                    "type": "file",
                    "url": "https://files.pythonhosted.org/packages/1f/d4/52869fe4ed24e373694c116adc00c82e5d92c747be4cbc97b24af43807f6/pyenchant-3.2.0.tar.gz",
                    "sha256": "6b97e9a9f132fa7c9029a6635d821ccad67d4980e68186d02c765b4256b6f152"
                }
            ]
        },
        {
            "name": "python3-PyPDF4",
            "buildsystem": "simple",
            "build-commands": [
                "pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"PyPDF4\" --no-build-isolation"
            ],
            "sources": [
                {
                    "type": "file",
                    "url": "https://files.pythonhosted.org/packages/4f/1f/509b44850c475c101aa5b5c9b81755cedd363389d6fbb5c53be6fa915a61/PyPDF4-1.27.0.tar.gz",
                    "sha256": "7c932441146d205572f96254d53c79ea2c30c9e11df55a5cf87e056c7b3d7f89"
                }
            ]
        },
        {
            "name": "python3-setproctitle",
            "buildsystem": "simple",
            "build-commands": [
                "pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"setproctitle\" --no-build-isolation"
            ],
            "sources": [
                {
                    "type": "file",
                    "url": "https://files.pythonhosted.org/packages/a1/7f/a1d4f4c7b66f0fc02f35dc5c85f45a8b4e4a7988357a29e61c14e725ef86/setproctitle-1.2.2.tar.gz",
                    "sha256": "7dfb472c8852403d34007e01d6e3c68c57eb66433fb8a5c77b13b89a160d97df"
                }
            ]
        }
    ]
}

You'll notice that there is no mention of PySide or Qt. The key module that the app depends so heavily is not there, why? Well because both Qt python dependencies are not available as source files from pip. Pip hosts source files and wheels. All the above are source files. These are tar balls that pip will compile and build from source. Wheels are, once again, pre-built binaries. Flatpak doesn't include wheels by default since it wants everything to be as cross platform and cross architecture as possible.

I also wanted to be as platform agnostic as possible so I attempted to build my python bindings from source in the flatpak. Flatpak did not like this. Neither PySide nor PyQt would build and each threw different error during the compilation process. My best guess would by the sandbox does not contain some required system libraries but I am at a loss to know which ones. And C compilers are not the most helpful thing as your only point of reference for a system error.

Lasers to the rescue

At this stage, I didn't actually know what python wheels were, I just assumed everything had to be built from source. I was desperate for answers after 3 days of debugging and staring at config files. I did something I very rarely do, I used the GitHub search function. To my amazement I found a solution.

The Optical Metrology Group in the Physics department of Humboldt University Berlin created a Linux application for locking onto spectroscopy lines. Guess what, they made a flatpak!

I looked at there yaml, and I knew then and there that they had found a solution.

# we used to build pyqt on our own here, but this build caused trouble with the pyqtgraph widget being transparent on some systems.
  # Therefore, we just use the x86_64 wheel here...
  - name: pyqt-x86_64
    buildsystem: simple
    sources:
      - type: file
        url: https://files.pythonhosted.org/packages/4c/bb/7fce18fbe0275d7a3e069a306d8f4662c77eda30ec6780634fd4a7ee50ce/PyQt5-5.15.1-5.15.1-cp35.cp36.cp37.cp38.cp39-abi3-manylinux2014_x86_64.whl
        sha256: b1ea7e82004dc7b311d1e29df2f276461016e2d180e10c73805ace4376125ed9
      - type: file
        url: https://files.pythonhosted.org/packages/31/24/f887203677955ba4d5d4efe9176ac7ed2bf84efce8c243ab91e63183ad9e/PyQt5_sip-12.8.1-cp37-cp37m-manylinux1_x86_64.whl
        sha256: a1b8ef013086e224b8e86c93f880f776d01b59195bdfa2a8e0b23f0480678fec
    build-commands:
      - pip3 install PyQt*.whl --target=/app/lib/python3.7/site-packages

They had obviously also faced similar Qt-based problems. Their YAML led to me finding out about python wheels, how I could get PyQt in a flatpak, and how I could tell pip the right directory for a python installation (I had no idea what --target would do before this).

Thanks to some German physicists, my next project is back on track. Beady eyed people may notice some information in some of the config files I've posted that hint to what it could be. I did try and use PySide using the same wheel method but I got into some strange cyclical versioning thing since it required PySide an Shiboken2 to be installed at the same time. All that means is that my application will be open source out of a legal obligation, as well as an civil one.

SIDE: kerberos is also a requirement for PyQt to work in a flatpak, I have no idea why but it may have something to do with certain networking modules.

Inspection and Dissection: Pepys - A Straightforward Markdown Editor

Mon, 31 May 2021 16:47:38 +0000

Inspection and Dissection: Pepys - A Straightforward Markdown Editor '/>

Inspection and Dissection: Pepys - A Straightforward Markdown Editor

The product page is available here

Brief Description

Pepys is a GUI journaling application built using Python bindings for the Qt framework. It is available as an installer for Windows and a Flatpak on Linux.

Background

I started journaling around February last year, it was probably the most prescient thing I have ever done. It wasn't long before we all went into self-imposed isolation and journaling provided a good way to express my thoughts.

My first posts were made in one big Word document. Journals need to be as future-proof as possible, and I thought at the time that since entire governments are probably reliant on MS Word that it won't be going away any time soon. It wasn't a good experience though. Having to type the date each time was tiresome, and I thought it would be a formatting mess should I use an Operating System without MS Word (something which would later happen when I spent several months using Linux).

I next moved on to Notion. An excellent productivity application that can be used to do all sorts from notes to spreadsheets to databases. It's template, tagging and calendar features made it a great prospect for writing my journal in. The entries were made in Markdown so would be viewable on any computer for as long as we still used ASCII and Unicode, seems pretty future proof. The trouble was, it made journal writing a chore. To write an entry I'd have to open up this electron application that took an age to open and slowly click through web links, waiting for pages to load, before I could get to writing entries. I wanted a native application that would allow me to write journal entries in markdown, with a date-based file chooser. I found no application that fitted all this criteria that worked on Linux and Windows, they all seemed Mac only (I am envious of the great ecosystem Apple cultivates among its development community). So I took it upon myself to write my own.

A Rocky Start

The first idea for a Journal application first came to me in June 2020 but it would take nearly a year before I would find the time and the courage to actually start it.

My only experience with a GUI application that wasn't made in a dedicated game engine was Dice-Jack which was a version of blackjack that used Dice instead of cards and was made over a few days for an assignment as part of my Sixth Form Computing course. I knew I couldn't use dotNet since I wanted it to be able to run on Linux as well as Windows. My next thought was Java, since that was the language I was learning at University at the time and seemed to fit all my requirements.

I managed to get a very early text editor working in Java. At that point, the idea was to have all the styling done in real time as you typed. For example, if I typed a # then I would set the font size you type in to that of a title.

I was running into issues that were perfectly fixable but I was never truly comfortable in Java. I will absolutely use Java in the future but it just didn't seem right with this project. With Java I had the choice between the ancient swing and JavaFX. I have never got to grasps with the whole split GUI framework where some bits are in code and some bits are in XML files. All the GUI frameworks are going this way so I will have to get my head around it eventually but this was not the project for this.

A Walk in the Dark

After bouncing of Java frameworks I had to go with what I know, Python. Python was the first language I learned and I've been using it for coming up 4 years now. The big players in the Python GUI game is Gtk, Qt, wxWidgets, and Tkinter. Tkinter and wxWidgets are the more simple of the four with the majority of applications being built in either Gtk or Qt. In the end I settled with Qt because it seemed to have better look and feel across different systems, and Gtk seemed to want me to use build systems that I was unfamiliar with. I first broke ground with the first git commit being pushed at 20:29 BST on April 1st. This was the first commit to the repo that would become the final release, this came at the end of several weeks of the aforementioned bouncing around.

Syntax Highlighting

An early version of the edit pane

My primary focus was in getting markdown source highlighting in the text window. Things like bold, italic, and strike-through came rather quickly but became incredibly slow with large amounts of text. My initial solution to the syntax highlighting was a naive one. I would be performing regex searches and applying text character formatting across the document on each key press. The approach was okay on small documents, but the larger the document the more text would be searched through and it would become impossible to type. The first breakthrough came when I discovered a Qt widget called QSyntaxHighlighter which provided methods that would allow text to be broken up into 'blocks'. I put all my regular expressions into a QSyntaxHighlighter and used its blocking mechanism to allow for typing to not be slowed down on large documents.

HTML Preview

An early version of the view pane

Alot of the early work went into refining my markdown regular expressions, making sure they formatted the correct parts and matched the output reasonably. It wasn't long before I had to turn my attention to the HTML preview. The preview pane was initially just a QTextBrowser that supported a limited subset of HTML. When the preview button was clicked the markdown would be converted to html using [mistune](https://mistune.readthedocs.io/en/latest/). I had used mistune previously on the [first iteration of this website](projects/inspection-and-dissection-this-site) so a lot of that could be copied and pasted from my work there.

The fact that only a subset of HTML was supported soon became an issue and I had to switch the preview to a full on web engine (the reason why the application is so large). The web engine was such a double edged sword. Without it: previews wouldn't match exports, math equations wouldn't be supported, many markdown elements couldn't be rendered properly (tables, inline html). But with it, the application swells in size to over 100MB download and even larger when fully installed. I wish I could resolve this alas I am too tired and too unskilled to be able to implement a solution.

The Killer Feature

It was around this time that I went into exploring pandoc. I was amazed by its power and I will undoubtedly utilise it in many future projects. Pandoc is a program written in haskell that uses its own flavour of markdown to provide a huge amount of outputs for document conversion. I knew I had to implement this in some way to provide some way of providing a wide range of export option. Then I will have truly realised my need of having entries been in a future proof format, why have 1 future proof format when you can have every format under the Sun?

Calendar File Selector

I knew that my main method of interacting with files should be through a calendar. Qt provides a reasonably good calendar widget that saved alot of time. The way I keep track of the files in a user's journal is by having a very rigid folder structure which Pepys gets pointed to. The journal directory has a structure of YYYY\MM\DD\YYYY-MM-DD\YYYY-MM-DD.md This makes it very easy to search for all the entries by date.

Release

A comparison of different logos I went through

I have went into my woes over releasing software here but it is safe to say that this is an are where I learned an awful lot. In the end Pepys was released on Linux via flatpak and on Windows via an NSIS installer available to download on GitHub. There was no Mac release because I do not own a Mac no have sufficient experience with Macs to be comfortable with creating installers and sufficiently testing them for correctness.

A Retrospective

This whole process has been the longest I have ever spent on a project, it was probably the largest project I have ever made, and there were alot of firsts. It was the first application I made that wasn't a game, it was the first proper GUI application I have made, and it was the first time I had to make installers by hand. The process was arduous at times and I felt like giving up at some points.

I sit here now after releasing it and I think "if I'd only done this" or "it would be so much better if..." but I can't think like that. If I had spent as long as possible on each feature till it was perfect then the software would never be released. I have learned that software development is an iterative process in 2 ways. The first way is in relation to an individual piece of software. You develop the software over time, slowly improving and adding until a finished product stumbles over the finish line. The second way is at a macroscopic level. Each huge project your begin, each journey you embark on, brings a new iteration on what went before. The next GUI project I make will be better. The next installer I make will be better. And this excites me.

I suppose it has taken me until the end of my first year of University to understand what learning is really about. Learning isn't about passing tests, although schools may tell you otherwise. Learning isn't even about knowing stuff, that is just a by-product. Learning is about being better, about knowing you will be better, and about being excited about that. I am excited about what I will create next, how I will iterate on what I have learned, and how I can surprise myself.

Special Thanks

All of this took an awful lot of effort so I would like to give special thanks to the following people and projects for there valuable insights.

Apostrophe : An excellent markdown editor that shows how to make a properly good writing experience and led me down the Pandoc path. Also inspired some of the regex used.

Humboldt University Optical Metrology Group : Without whom I'd still be wrestling with compilers in Flatpak

Lucy Marsden (Github) : Who provided a second set of eyes and motivated me to actually get stuff done.

Jan Grulich : For showing how to implement Adwaita colours in Qt

Full Release: newcastle-bst - Harvard referencing style as recommended by Newcastle University

Sat, 10 Jul 2021 10:33:04 +0000

Full Release: newcastle-bst - Harvard referencing style as recommended by Newcastle University '/>

Full Release: newcastle-bst - Harvard referencing style as recommended by Newcastle University

A couple of months ago I released a blog post about LateX. At the bottom of the post I made quick reference to a BibTeX style I had created for the referencing used by Newcastle University. The style had a few bugs and was quite hard to find it but it was left dormant while I made some other software and focused on work.

I am now pleased to say that I have spent some work polishing it up, fixing some issues, and giving it a proper release. It's even on CTAN!. If you are a student and notice any issues, please report them on GitHub

Release Notes

newcastle-bst: Harvard referencing style as recommended by Newcastle University

This package provides a BibTeX style to format reference lists in the Harvard at Newcastle style recommended by Newcastle University. It should be used alongside natbib for citations.

Installation

The required style file is available from GitHub and CTAN. You can use the style by copying it into your working directory containing your .tex file. You can also add it to your bst directory in your tex path to use it without having to copy it over each time.

Using the style

To use the style include this in your preamble:

\usepackage{natbib}
\usepackage[UKenglish]{isodate}
\bibliographystyle{newcastle}

Also remember to specify your .bib file at the end of the document:

\bibliography{file}

The easiest way to create .bib files for this style is through exporting entries from a reference manager such as Mendeley. However, some parts are not available through this (such as titleaddon for computer programs). If you notice any discrepancies between generated references and the recommended styles then please raise this on GitHub

License

The text files contained in this work may be distributed and/or modified under the conditions of the LATEX Project Public License (LPPL), either version 1.3c of this license or (at your option) any later version.

This work has had no input from Newcastle University and is done entirely in order to help other students create bibliography quicker.

This work is ‘maintained’ (as per LPPL maintenance status) by Luke Briggs.

Inspection and Dissection: Calmer Internet - My First Web Extension

Wed, 18 Aug 2021 15:18:16 +0000

Inspection and Dissection: Calmer Internet - My First Web Extension '/>

Inspection and Dissection: Calmer Internet - My First Web Extension

The product page is available here

Brief Description

Calmer Internet is a web extension for Chrome, Firefox, and Edge that makes the internet a less infuriating place to be. It removes comments, recommended content and various trending pages from Twitter, YouTube, and Instagram to try and relieve that doom-scrolling.

Rationale

Since I finished stage 1 of University in June, I had a considerable amount of time off. A lot of that time was spent in a state of not quite working and not quite relaxing. I was stuck in a cycle of following the YouTube sidebar of bland content before going to Twitter and scrolling down the trending hashtags and comments, getting annoyed at every step.

Many social media platforms just throw various things at you in an attempt to just keep scrolling and clicking and engaging, before you know it you've spent an hour of your life on a soul-sucking site.

I didn't want to quit these sites entirely. There are truly incredible and insightful people on there like Tom Scott, exurb1a, and Ahoy. Twitter is also good as a micro-blog feed for people I want to follow in the tech sector. The conclusion I came to is that I need to take control of what I see. I should only be consuming things I have actively sought out, because those are the things that I enjoy the most. It also means that I can't just keep scrolling and watching forever, I will run out of content because I only see what I have subscribed to. There is no 'one more click'

My initial solution was to have everything go to an RSS reader but most social media platforms don't play nicely with rss feeds and feed readers don't really play nicely with conveying threads, retweets and video content. As a result, I turned to editing the sites themselves to provide a more focused experience.

How it's made

This extension didn't start out as an extension. It started out as a set of rules for my ad-blocker. I'd use the eyedropper tool to select the Trending bar on Twitter just so I wouldn't be tempted to click it. I did similar things with the trending tab on YouTube. It then got to a point where I wanted to do more like remove the recommended sidebar and the video-wall end screens, so I decided I needed a web extension.

I have never made a web extension before, but I do have some experience in HTML and JavaScript so it wasn't too troublesome to get used to. Most of what my extension does takes place in a 'content script'. This is a javascript file that just loaded alongside the current page at load time. The file then looks at the current URL and checks the page for any elements I've told it to remove.

// Adding Twitter elements to a list of elements to remove
    if (window.location.hostname === "twitter.com" || window.location.hostname === "mobile.twitter.com"){
        if(!settings["TwitterTrendingBar"]){
            elementsToRemove["trendingBar"] = document.querySelectorAll("div[aria-label='Timeline: Trending now']")
        }
        if(!settings["TwitterExploreLinks"]){
            elementsToRemove["exploreLinks"] = document.querySelectorAll("a[href='/explore']")
        }
        if(!settings["TwitterWhoToFollow"]){
            elementsToRemove["miscStyling"] = document.getElementsByClassName("css-1dbjc4n r-1867qdf r-1phboty r-rs99b7 r-1ifxtd0 r-1bro5k0 r-1udh08x")
            elementsToRemove["whoToFollow"] = document.querySelectorAll("aside[aria-label='Who to follow']")
        }
        if(!settings["TwitterTopics"]){
            elementsToRemove["topics"] = document.querySelectorAll("div[aria-label='Timeline: ']")
        }
    }

The script then attaches a listener that causes new checks every time the webpage is updated

function observeBodyRemove(){
        // Select the node that will be observed for mutations
        var targetNode = document.querySelector('body');

        // Options for the observer (which mutations to observe)
        const config = { attributes: true, childList: true, subtree: true };

        // Callback function to execute when mutations are observed
        const callback = function(mutationsList, observer) {
            elementsToRemove = getElementsToRemove();
            for(key in elementsToRemove){
                for (element of elementsToRemove[key]){
                    try {
                        element.remove();
                    }
                    catch(e){
                    }
                }
            }
        };

        // Create an observer instance linked to the callback function
        const observer = new MutationObserver(callback);

        // Start observing the target node for configured mutations
        observer.observe(targetNode, config);
    }

The only other JavaScript in the extension is in a 'background script'. Whereas 'content scripts' are loaded alongside a webpage, background scripts are persistent and always run whenever the extension is loaded. This means they are useful for doing things before a page is loaded. One feature of the extension is you can redirect any links to YouTube.com to youtube.com/feed/subscriptions, so you aren't bombarded by useless recommendations. I initially had the redirect in a content script but that required the whole page to be loaded before redirecting away from it.

My current solution uses the WebRequest API to intercept a page request and load a new one instead. This is also done be setting up a listener, such as this one:

// YouTube
    chrome.webRequest.onBeforeRequest.addListener(
        function (details) {
            if(!settings["YTHomeRedirect"]) {
                return {redirectUrl: "https://youtube.com/feed/subscriptions"};
            }
        },
        {
            urls: [
                "*://*.youtube.com/"
            ],
            types: ["main_frame", "sub_frame", "stylesheet", "script", "image", "object", "xmlhttprequest", "other"]
        },
        ["blocking"]
    );

Finally, all that was left was to create a manifest. I have great experience with manifests since Pepys recently received builds for rpm, .deb, and PKGBUILD.

{
  "manifest_version": 2,
  "name": "Calmer Internet",
  "version": "1.1.0",
  "browser_specific_settings": {
    "gecko": {
      "id": "calmerinternet@lukebriggs.dev"
    }
  },
  "browser_action": {
    "default_icon": {
      "16": "icons/16.png",
      "24": "icons/24.png",
      "32": "icons/32.png"
    },
    "default_title": "Calmer Internet"
  },
  "icons": {
    "48": "icons/48.png",
    "96": "icons/96.png"
  },
  "content_scripts": [
  {
    "matches": [
      "*://*.youtube.com/*",
      "*://*.twitter.com/*", "*://twitter.com/*",
      "*://*.instagram.com/*"
    ],
    "js": ["calmer.js"]
  }
  ],
  "background": {
    "scripts":["background.js"]
  },
  "permissions": [
    "*://*.youtube.com/*",
    "*://*.twitter.com/*", "*://twitter.com/*",
    "*://*.instagram.com/*",
    "storage",
    "webRequest",
    "webRequestBlocking"
  ],
  "description": "Provides a calmer internet experience. Removes comments and various recommendation elements from Twitter, YouTube, Instagram.",
  "options_ui": {
    "page": "options.html"
  }
}

Deployment

The extension is available on the Chrome Web Store, the Firefox Add-on Store, and the Microsoft Store. WebExtensions are now pretty cross-compatible, so I use the exact same code-base across all browsers. It would even support Safari as of Big Sur, but I do not have a Mac to upload it, nor do I want to fork out the $99 price of an Apple Developer account.

As ever, the project is also available in full on my GitHub

Inspection and Dissection: Sounds2Spotify

Thu, 16 Sep 2021 15:07:04 +0000

Inspection and Dissection: Sounds2Spotify '/>

Inspection and Dissection: Sounds2Spotify

Available on Github

Brief Description

Sounds2Spotify is a web extension for firefox and chrome that converts the tracklists that appear on BBC Sounds programme pages to be converted into Spotify playlists.

Unlike Calmer-Internet this extension can only be installed by following the instructions on Github due to its use of API keys.

Impetus

One of my pleasures in life is listening to Music. My favourite tracks are stored locally and also in Spotify playlists for when I'm using a device without my collection. Most of the time, however, I'm not listening to my favourite tracks, I'm listening to the BBC 6Music. More specifically, I'm listening to the Lauren Laverne Breakfast Show.

Due to licensing reasons the BBC can only have its music programming available on its own BBC Sounds app for a limited time. This is fine when I'm using my computer or phone, but quite often I want to listen to music in the background when using my PlayStation, which only allows music to be played through Spotify.

The goal: to convert the music played on BBC Sounds to Spotify playlists.

Code Walkthrough

Part 1: Getting the Music

Thankfully, underneath every music programme, the BBC provides a track list of all the music played throughout the programme as well as a popover menu providing links to external sources.

So the first job my extension had to do was to scrape this information. There are no APIs or direct links to this information so the only way to get this information is by clicking on each button and grabbing the Spotify link.

// List of buttons that bring up track popover
var list = document.querySelectorAll('[id^="track-"]')
var tracklist = {}
var episodeDate = document.getElementsByClassName("sc-c-episode__metadata__data")[0].lastChild.data

Each menu button has the HTML ID of "track-x" where x is its position in the tracklist. So a query for all these IDs will yield a list of button elements.

The date the episode aired is also quite useful when it comes to naming the eventual playlist, so I grab that as well.

I also initialise the tracklist here. Eventually I want a list of the Spotify links for the tracks.

The IDs and element classes were obtained by hand by scouring the HTML source of the page.

for (var i=0; i < list.length; i++){
    list[i].scrollIntoView()
    list[i].click()
    await sleep(t) // t = 200ms
    try{
        var track = document.querySelectorAll('[href^="https://open.spotify.com/track/"]')[0]
        var trackMetadata = JSON.parse(track.getAttribute("data-bbc-metadata"))
        var trackTitle = trackMetadata.TID
        tracklist[trackTitle] = track.getAttribute("href")
    } catch {
        // No links found
    }
}

I loop through all the button elements, scroll to the element and click it. There is a 200ms delay between each track as render differences can lead to buttons not being clicked.

When a track is clicked, single popover appears containing a Spotify track link if it exists. As a result, by querying for a Spotify track URL on the page, I will get the link, if it exists.

The popover also includes some metadata for the track in the form of JSON. So I use the track title as the key in a dictionary of tracks, just to make debugging easier. The value in the dictionary is the corresponding link.

chrome.runtime.sendMessage({type: "tracklist" , track_list: tracklist, name: document.title, date: episodeDate}, function (response) {
    console.log(response.farewell);
})

All this data is then wrapped up into a message and sent off to be picked up by a background script.

I mention background scripts in more detail on my Inspection and Dissection on Calmer-Internet

Part 2: Promises, Async, APIs

Now for the complicated part. Using the list of Spotify links I have to create a playlist.

// Base URL
get_url = "https://accounts.spotify.com/authorize?"
get_url += `client_id=${client_id}&`
get_url += "response_type=code&"
get_url += `redirect_uri=${encodeURIComponent(chrome.identity.getRedirectURL())}&`
get_url += "scope=playlist-modify-public"

The first thing I need to do is get authorisation from the user to allow the extension to modify public playlists. This is done by pointing a request to a specific url containing certain information.

client_id: this is and id associated with a Spotify app.
response_type: there are various ways to authenticate, the way I are doing it is through a 'code' flow
redirect_uri: the url Spotify should redirect to when done.

Web extensions require this to be a single link unique to the extension which is obtained through the method above

scope: which permissions I require

chrome.runtime.onMessage.addListener(
    function(request, sender, sendResponse) {
        if (request.type == "tracklist"){
            sendResponse({farewell: "recieved tracklist"});

            tracklist = request.track_list
            tracklist_name = request.name
            tracklist_date = request.date

            // Authenticate and make playlist
            chrome.identity.launchWebAuthFlow(
                {
                    url: get_url,
                    interactive: true,
                }, authenticateSpotify)
        }
    }
);

This is a listener that responds to my message containing a tracklist. All the listener does is store the information in global variables (I know it can be bad practice but this is a small program with a specific purpose) and launch an authentication flow with my link.

Once this is done, I then call the authenticateSpotify method.

function authenticateSpotify(response){
    // Get parameters from response
    var urlParams = new URLSearchParams(response.replace(chrome.identity.getRedirectURL(), ''))
        //...
}

My WebAuthFlow passes a response in the form of a URL containing a parameter with an authorisation code (or a rejection). I isolate these parameters by stripping off My redirect url and calling URLSearchParams on it.

// POST request for access token
    var authRequest = new Request("https://accounts.spotify.com/api/token",
    {
        method: "POST",
        headers: {
            'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'
        },
        body: new URLSearchParams(
        { 
        'grant_type': 'authorization_code', 
        'code': urlParams.get("code"), 
        'redirect_uri': chrome.identity.getRedirectURL(),
        'client_id': `${client_id}`,
        'client_secret': `${client_secret}`
        }) 
    })

I then construct a POST request to Spotify by exchanging my authorisation code, and client details for an access token.

// Create playlist
    fetch(authRequest).then(function(response){
        response.json().then(function(json){
            createPlaylist(json.access_token)
        })
    })

I then send this POST request, receive a response, parse the JSON response and send the access token to the createPlaylist function.

async function getSpotifyID(access_token){
    var userRequest = new Request("https://api.spotify.com/v1/me",
    {
        method: "GET",
        headers: {
            Authorization: `Bearer ${access_token}`
        }
    })
    const response = await fetch(userRequest)
    const json = await response.json()
    return json.id
}

To know which account I'm working with I ask for the profile ID.

async function createPlaylist(access_token){
    var trackURIS = []

    // Add create Spotify URIs from track links
    for (var track of Object.values(tracklist)){
        trackURIS.push(`"${track.replace("https://open.spotify.com/track/", "spotify:track:")}"`)
    }
}

The Spotify API does not want the tracks in the form of a link it wants a uri in the form spotify:track:trackid. So for each link in our track list I replace the link part with spotify:track:

// Create new playlist
    var request = new Request(`https://api.spotify.com/v1/users/${id}/playlists`,
    {
        method: "POST",
        headers: {
            "Authorization": `Bearer ${access_token}`,
            "Content-Type": "application/json"
        },
        body: 
`{
    "name": "${tracklist_name} | ${tracklist_date}",
    "description": "Made using Sounds2Spotify"
}`
    })

Before I can add to a playlist, I need to create one first, so I construct a request to do just that using the authorisation token, and the user ID. I use the page name, and date from the BBC Sounds page as the title of the playlist.

// Add tracklist songs to newly created playlist
    var request = new Request(`https://api.spotify.com/v1/playlists/${playlistID}/tracks`,
    {
        method: "POST",
        headers: {
            "Content-Type": "application/json",
            Authorization: `Bearer ${access_token}`
        },
        body: 
`{
    "uris": [${trackURIS}]
}`
    })

    var response = await fetch(request)

The response of my request contains the playlist ID which I then feed into our next request to add my list of tracks to the playlist.

// Open tab to new playlist
    chrome.tabs.create({ url: `https://open.spotify.com/playlist/${playlistID}`})

Finally, I open up the playlist in a new tab.

Retrospective

Overall, I am very pleased with what I managed to achieve. I have very limited experience in using JavaScript to interact with REST APIs, but I managed to pick it up reasonably quickly.

The code I think is relatively clean for a first attempt. Obviously if any of the request throw an error then there is not much in the way of handling a response, but JavaScript tends to be pretty robust with errors and this isn't a product that is going to be released on any web store or anything.

Reason for Installation Procedure

The reason why this project is not going to be released on a web store is for security reasons. A part of the process requires sending Spotify secret API keys for the corresponding Spotify app. This would mean that my secret keys would have to be in the client-side code, and therefore be freely accessible.

It would therefore be possible for someone to pose as my extension and start making requests as my app to users.

The installation process isn't so cumbersome to be impossible to install, but it isn't as plug and play as I would like it to be

Sources of Information

Spotify API Docs - https://developer.spotify.com/documentation/web-api/
- Playlists
Chrome extension API Docs - https://developer.chrome.com/docs/extensions/reference/
- identity
- Message Passing
Mozilla Web API Docs - https://developer.mozilla.org/en-US/docs/Web/API
- fetch()
- Response
- Promise

Shiny New Things

Thu, 11 Nov 2021 15:48:20 +0000

Shiny New Things '/>

Shiny New Things

New Hardware

The RISC revolution claims a new victim. On October 25th I became the owner of my first Apple computer, my first non-x86 desktop, and what is already the best computer I have ever used.

Past devices

I moved away from Windows about 9 months ago and I really haven't looked back. A few things changed in that gestation period. Firstly, I settled on Fedora 34 as my final distro: GNOME 40 made some changes I liked and also my bluetooth audio devices worked better. Secondly, and more importantly, my priorities shifted. September saw the return of in-person University lecturing and some changes had to be made.

My previous computer was a Lenovo Legion 5, it had a efficient (but still power hungry) Ryzen 5 4600h, and a dedicated Nvidia GPU. The laptop was 15.6", weighed 2.4kg and would require obnoxious fans to spin up if it even thought you might do something demanding. The laptop was a symptom of being purchased at the beginning of stay-at-home orders by someone who overestimated the amount of games they would be playing.

When it came to in-person teaching, it also was a bit cumbersome to use the machines on-site. At the beginning of the term, the University decided move away from using local software to having Azure Virtual Desktops instead, with the PCs acting as thin clients. Even with gigabit networking and decent hardware, virtual desktops still have a strange smell to them. The main issue was the fact that they are still in the teething phase and IT departments are not quick to react. Not every virtual desktop has exactly the same software, and there is no way to know which desktop you are going to get. One day I logged onto a virtual desktop and it had all the software I needed except git. My only solution was installing a portable version of git into my OneDrive so I could guarantee a git install on each machine. Virtual desktops also made python virtual environments a hassle since every time I would do work on my personal machine, all the symlinks from the AVD would be broke and I'd have too recreate it.

So here were my new requirement:

A CPU at least as powerful as my previous laptop
16GB RAM (8GB was too little when doing development alongside local virtual machines)
No dGPU (the weight, battery, and noise overhead is just awful)
Screen larger than 13", ideally 14.5 - 15"
Portable, ideally around 1.5kg

Like-to-haves:

120Hz screen (Legion 5 had it and it made things very nice)
512GB hard drive (Legion 5 had 256GB and I was pushing up against it)
A iGPU good enough to play basic 3d games at full speed

There was a selection of PC laptops that met some of my requirements. However, a lot of them put the higher RAM behind the models with a dGPU, which is an instant turn-off.

The incredible performance of last year's M1 chips got my head pointed towards Cupertino. They were efficient, quiet, had a strong GPU, blazing CPU and were bundled with a first-party Unix OS. Last year's MacBooks did not meet my requirements for the case however. A 13" screen is too small to be doing comfortable development work on.

So I sat eagerly awaiting the November Apple Event, hoping that rumors of a more powerful 14" MacBook Pro was on the horizon.

The announced computers met every single one of my criteria, the price was steeper than I had been anticipating but I had some disposable income and I had fallen slightly in love.

The Mac Factor

I took delivery of my base-model 14-inch MacBook Pro (M1 Pro 8/14) on day one, and have used it every day in the 2 weeks since. The true reason why I went slightly over-budget is due to the fact that these computers are the first in a while to be truly exciting to me. A whole new instruction set platform in the desktop space is not something that has happened in my computing lifetime and it just awakens the computer nerd inside of me. Speed and battery life, graphics and little fan noise, to mention nothing of the fact that Apple devices have the most high-quality feel of any laptop.

The software enthusiast inside me also loves the idea of being one of the first on a new platform. Not all the open source apps I use have been compiled for ARM yet, I could be the one to do it! Heck, I have my own apps to port!

New Site

If you have a look around you may also notice a slight redecoration. This is iteration ⅠⅠⅠ of my site.

The first was a dynamic site served through my own flask web server,
The second was a static site generated through Hugo,
This iteration is a static site generator of my own devising

I made the change because my last site looked pretty cluttered (and I felt that my design skills have improved since my first attempt). I also wanted a nicer way to list my projects.

The process of developing a static site generator in python was much more straightforward than I thought. It only took around 4 days to get to a point where everything looked nice on the outside. The codebase needs some cleanup, and it isn't entirely ready for anyone other than me to use it, but it works.

The program on initialisation creates a few directories to place stuff (content, projects, static, etc) as well as a config file for certain things like the navigation bar. The built website gets put into a public directory. The building process itself starts by copying the static directory tree into the public directory. A python library called mistune converts the markdown files in content (along with YAML metadata headers) to HTML, pygments is also used for syntax highlighting. The converted html posts are then passed to my custom jinja2 templates that have all the styling and base markup. There are a couple of other things like auto-generated social links on the homepage and a so-basic-its-almost-broken built-in HTTP server for viewing webpages locally.

I will probably tweak a few things over time, but I think I've settled on the principle design, at least for a while.

The SSG is named Victor as a tongue-in-cheek nod to Hugo, I also quite like that it is also the name of Mr. Freeze which kind of makes sense for a static site generator.

The source code is available on GitHub

Memory in Assembly - How different areas of memory are treated at the metal

Fri, 20 May 2022 17:57:44 +0000

Memory in Assembly - How different areas of memory are treated at the metal '/>

Memory in Assembly - How different areas of memory are treated at the metal

I thought it might make for an interesting learning exercise to see how memory in different areas is treated at the assembly level. The program itself is simple, but I hope it will provide some insight.

C Program

const int constant_int = 0xc07;
int main() {
    static int static_int = 0x57a71c;
    int stack_int = 0x57ac3;
    int* malloc_int = malloc(1);
    malloc_int[0] = 0xa1;
    func_int(stack_int);
    func_int(*malloc_int);
    func_int(static_int);
    func_const(constant_int);
}

ARMv8 Assembly

stp         x29,x30,[sp, #local_20]!                
mov         x29,sp                                  
mov         w0,#0x7ac3                              
movk        w0,#0x5, LSL #16                        
str         w0,[sp, #stack_int+0x20]                
mov         x0,#0x1                                 
bl          __stubs:__stubs::_malloc                ;void * _malloc(size_t param_1)
str         x0,[sp, #malloc_int+0x20]               
ldr         x0,[sp, #malloc_int+0x20]               
mov         w1,#0xa1                                
str         w1,[x0]                                 
ldr         w0,[sp, #stack_int+0x20]                
bl          func_int                                ;void func_int(int x)
ldr         x0,[sp, #malloc_int+0x20]               
ldr         w0,[x0]                                 
bl          func_int                                ;void func_int(int x)
adrp        x0,0x100008000                          
add         x0,x0,#0x0                              
ldr         w0,[x0]=>__data:main::static_int        ;= 57A71Ch
bl          func_int                                ;void func_int(int x)
mov         w0,#0xc07                               
bl          func_const                              ;void func_const(int x)
mov         w0,#0x0                                 
ldp         x29=>local_20,x30,[sp], #0x20           
ret

Static Integer

You may notice that our static (0x57a71c) variable is not declared in the assembly. This is because everything you see above is in the TEXT segment. The part of the assembly that is read only. Text segment is labelled as such with an assembly directive.

Static variables exist for the lifetime of the program but can change their value and so shouldn't be loaded into the text segment. Instead, values that can be edited but are static and initialised are loaded into the data segment.

// __data
// __DATA
// ram:100008000-ram:100008003
//

_static_int.0
main::static int
    int     57A71Ch
//
//__DATA
//__DATA
// ram:100008004-ram:10000bfff

With Ghidra we can see that our file contains a directive to load our static variable into the data segment at launch. We also have a label to this static variable that somewhat matches our C code.

Stack Integer

This works by moving our value into a register, then storing the contents of that register into an area of memory offset from the stack pointer. The stack pointer is a special register that the CPU uses to tell us where our stack is. Because everything we store on the stack is a known size, the compiler works out the offsets and knows where things should be stored in relation to each other.

C

int stack_int = 0x57ac3;

Assembly

mov         w0,#0x7ac3               ; move integer into register 0                             
movk        w0,#0x5, LSL #16                        
str         w0,[sp, #stack_int+0x20] ; store integer into RAM address pointed to by
                                     ; stack pointer (sp) + an offset based on number
                                     ; size. Our number is now in stack memory

Heap Integer

This works by asking for a given amount of memory to be allocated, the kernel allocates the new memory in our address space then hands us back a pointer to it. The address we get back is stored on the stack, and we can load values into this address to store on the heap.

C

int* malloc_int = malloc(16);

malloc_int[0] = 0xa1;

Assembly

mov         x0,#0x1                     ; Set first parameter of malloc to 1 (number of 
                                        ; bytes to allocate)                        
bl          __stubs:__stubs::_malloc    ; call malloc
str         x0,[sp, #malloc_int+0x20]   ; Store return value of malloc (register 0)
                                        ; onto the stack (sp + offset)




ldr         x0,[sp, #malloc_int+0x20]   ; Load heap address (which is stored on stack)
                                        ; into register              
mov         w1,#0xa1                    ; move our integer into register 1              
str         w1,[x0]                     ; Store our integer into the heap address 
                                        ; (integer is now on the heap)

Notice how the bottom two lines are effectively the same as what we did on the stack, everything else is all the extra overhead of using the heap, this is one of the reasons why it's easier to use the stack to store local variables (another obvious one is we don't have to free things on the stack whereas in a real program we would also have to remember to free this memory)

Calling a function with a stack variable

C

func_int(stack_int);

Assembly

ldr         w0,[sp, #stack_int+0x20] ; Set parameter 0 to value stored in the 
                                     ;  address pointed to by stack_int                
bl          func_int                 ; call function

Calling a function with a heap variable

C

func_int(*malloc_int);

Assembly

ldr         x0,[sp, #malloc_int+0x20] ; load the value stored at the malloc_int variable (our
                                      ; heap pointer) into register 0
ldr         w0,[x0]                   ; load the value at our heap address (the actual number) 
                                      ; into register 0 as the parameter to our function.                         
bl          func_int                  ; call function

Calling a function with a static variable

C

func_int(static_int);

Assembly

adrp        x0,0x100008000   ; Since static variables are in the data section,
                             ; we know the precise address (relative to our 
                             ; program) of them, and so we don't need to use the 
                             ; stack pointer, just the program counter which adrp 
                             ; does by default                        
add         x0,x0,#0x0                                    
ldr         w0,[x0]          ; load the value at the data address from memory 
                             ; into register 0 (our function parameter)
bl          func_int         ; call function

Calling a function with a constant variable

func_const(constant_int);

mov         w0,#0xc07   ; A constant variable never changes so we don't need to 
                        ; use addresses, the constant is here in the text section 
                        ; with the code and is loaded by value into register 0
bl          func_const. ; call function

Recap

Using stack variables, we load everything relative to the stack pointer.
With heap variables we first load the address we have on the stack, then load the value from this address
With a static variable we load everything from a known location relative to the program counter (in the data segment)
With a constant variable we can just use the values as is which is stored in the text of the assembly itself.

Painting by the Numbers: An exploration into evolutionary algorithms

Sat, 28 May 2022 17:05:11 +0000

Painting by the Numbers: An exploration into evolutionary algorithms '/>

Painting by the Numbers: An exploration into evolutionary algorithms

In a recent second year University project, I was tasked with developing an evolutionary algorithm for reproducing images using a limited number of semi-transparent polygons. I spent so much time above and beyond what I needed to that I thought it was worth writing a post about how I went about my work and challenges along the way.

What are evolutionary algorithms?

The basic premise of an evolutionary algorithm is: you have a population of solutions; this population goes through some sort of mutation, reproduction, or mating; the individuals in the population are then evaluated according to a particular fitness function and the fittest survive. The idea with evolutionary algorithms, as with evolution of biological species, is that given sufficient time, variation, and evolutionary pressure, it is possible for individuals to emerge that solve the particular problem you have placed upon it.

Just as when faced with the problem of getting food from land, some fish became amphibious, and eventually exclusively land creatures. Given the problem of wanting to look like the Mona Lisa, a population of shapes will emerge that solve this problem.

The brief

The brief was inspired by Roger Alsing's blog post where he did a similar thing. His algorithm was quite simplistic in what sort of mutations it offered and it was therefore felt we could make some improvement on this.

The final submission for the project was written in Python using the DEAP library, however, I fancied a speed-up. And using a compiled solution like PyPy was not cooperative and so I rewrote the program in Go. Because of various hoops and hurdles in releasing University coursework submissions, the original python implementation is not available. My Go rewrite, however, is available at my GitHub. I am not as experienced in developing in Go as with python but I feel I have made (mostly) sane, if not idiomatic, choices.

The Program

Making the population

var individuals []Individual
for i := 0; i < 256; i++ {
    individuals = append(individuals, makeIndividual(MAX_POLYGONS))
}

I start off by making the population of Individuals. In this case I make 256 individuals with each individual consisting of 100 polygons.

Our Polygon and Point structs looks like this:

type Polygon struct {
    color    rgba
    vertices []Point
}

type Point struct {
    x float64
    y float64
}

To make a polygon I simply create instances of these classes with various random starting values in order to provide a diverse population from which a solution can emerge.

func makePolygon() Polygon {
    // Create colour
    polygon := Polygon{color: rgba{
        r: uint8(rand.Intn(255)),
        g: uint8(rand.Intn(255)),
        b: uint8(rand.Intn(255)),
        a: uint8(rand.Intn(255-75) + 75),
    }}

    sides := 3
    // Create vertices
    for i := 0; i < sides; i++ {
        x := rand.Intn(200)
        y := rand.Intn(200)
        polygon.vertices = append(polygon.vertices, Point{x: float64(x), y: float64(y)})
    }
    // Arrange vertices such that they don't create a shape that self intersects if a polygon has more than 3 initial sides
    polygon.removeSelfIntersect()

    return polygon
}

See This section for more details on how I made sure polygons didn't self-intersect.

The polygons then go on to form an individual:

func makeIndividual(n int) Individual {
    var polygons []Polygon
    for i := 0; i < n; i++ {
        polygons = append(polygons, makePolygon())
    }
    ind := Individual{
        elements: polygons,
        fitness:  -1,
    }
    return ind
}

Individuals are initialised with a fitness of -1, which means that fitness is invalid and needs to be re-calculated. In order to prevent doing more processing than necessary, I will reset the fitness back to -1 whenever a change is made to that individuals polygons. Doing this means that I can guarantee that if a fitness is not -1, the polygons haven't change and I can skip that individual's evaluation.

Generating offspring

Every generation, new offspring are created from our current population. The strategy I went with (varOr) leads to each child being either a copy of a parent, a mutation of a parent, or a mating between two parents.

for i := 0; i < generations; i++ {
    offspring := varOr(population, 120, 0.25, 0.4)
    // ...
}

Mating involves getting a taking a random point in the list of polygons as a pivot point, then swapping everything after the pivot point with the same set of points from the other parent. Leaving a child where the first half(ish) of the polygons are from one parent, and the other half(ish) are from the second parent. Any children from this process are also given an initial fitness of -1, as the parent fitness is no longer valid due to half the polygons being different.

// Get pivot
cxpoint := rand.Intn(size-2) + 1
// Swap ends
temp := ind1[cxpoint:]
ind1 = append(ind1[:cxpoint], ind2[cxpoint:]...)
ind2 = append(ind2[:cxpoint], temp...)
// return children
return ind1, ind2

Reproduction involves a simple copying of one individual across generations, as such the fitness does not have to be invalidated to -1 because it is a perfect copy of an Individual with valid fitness.

// Choose individual
idx := rand.Intn(len(population))
// Clone individual's polygons
ind := clonePolygonSlice(&population[idx].elements)
// Add new individual with same polygons to offspring
offspring = append(offspring, Individual{elements: ind, fitness: population[idx].fitness})

Mutation is where a large amount of the uniqueness of this implementation comes in, and where a lot of the improvement was made during development. The following mutations can occur to an individual:

Change colour of polygon
Change transparency of polygon
Add a polygon to an individual (within 100 limits)
Remove polygon from an individual (within 20-100 limits)
Add a point to a polygon (requires removing self intersect)
Re-order how polygons are stacked
Change position of a single point
Change position of all points at once

The mutations were picked through a process of 1% inspiration, 99% calculation. With many trial and error runs of various mutations and algorithm choices before settling on the current set with their current probabilities of occurrence.

Offspring evaluation

Once we've got our children, we need to see how good they are. It is at this point that Go slightly comes into its own. Evolutionary algorithms, particularly the evaluation, are incredibly well suited to multithreading. Each evaluation is entirely independent and, as such, each one can run in its own thread, speeding up things significantly on multi-core systems.

It is also in this aspect where Go also comes into its own, as Go has first-class support for multithreading. If you have a thread-safe function, then spinning it up in a new thread is as easy as putting the keyword go before it.

var waitGroup sync.WaitGroup
for offIdx := 0; offIdx < len(offspring); offIdx++ {
    if offspring[offIdx].fitness == -1 {
        waitGroup.Add(1)
        go func(offIdx int) {
            defer waitGroup.Done()
            offspring[offIdx].fitness = evaluate(offspring[offIdx].elements)
        }(offIdx)
    }
}
waitGroup.Wait()

In this instance, we loop through all the offspring, and if the fitness is invalid (like it is the result of a mutation or mating) we evaluate the fitness of the Individuals polygons. All the WaitGroup stuff around it just holds a counter of how many routines are left to finish until we can carry on with the program. This allows us to utilise all the physical cores on our machine during the evaluation stage.

The evaluation itself is quite straightforward. We draw the individual out onto a canvas, convert it and the target image into greyscale, and then go pixel by pixel to calculate how different the image is. The individual with the smallest difference between its greyscale and the target greyscale is determined to have the highest fitness.

func evaluate(solution []Polygon) float64 {
    img := drawSolution(solution)
    diff := ImageDifference(img.Image(), TARGET)
    MAX := float64(math.MaxUint16 * img.Height() * img.Width())
    return (MAX - diff) / MAX
}

Offspring selection

After we have our children, we must pit them against each other to the death. The selection method I chose was tournament selection, this works be getting a random sample from the population and carrying the best from that sample into the future generation.

func TournSel(individuals []Individual, k int, tournsize int) []Individual {
    var chosen []Individual
    for i := 0; i < k; i++ {
        var aspirant []Individual
        for j := 0; j < tournsize; j++ {
            aspirant = append(aspirant, individuals[rand.Intn(len(individuals))])
        }
        chosen = append(chosen, BestSel(aspirant))
    }

    return chosen
}

I then draw out the best child each generation just so I can see the progress as it goes on. Apart from that, our generation is over and the cycle starts over again.

The results

Even though we never do anything other than random, indiscriminate changes, after about 250,000 generations we can get pretty good images made up of very few polygons.

Appendix

Non self-intersecting polygons

When it comes to the vertices of a polygon, you can't just start drawing lines between them and hope for the best.

If I have 4 points that are going to make up my square:

And then I start drawing lines between the points in any order:

That ain't no square chief. So, how can we tell which order we should connect the points? One way to do it is with the power of vectors and angles. First we start out with a base vector, which we just use to be the vector between the center of the polygon and the first point:

We then get vectors between the center and each point in the shape:

If we then get the angle between the base vector and all of those vectors, we notice a pattern:

The order in which we wan't to connect the dots in, also make increasingly large angles with the base vector. Therefore, if we get our points and sort them by the angle their vector makes with the base vector, they will be in the order in which we can draw them.

The code for the mathematics itself can be found in polygon.go

Playing Soldiers: An Exploration into Wartime Ciphers - Enigma and Lorenz

Mon, 29 Aug 2022 15:27:07 +0000

Playing Soldiers: An Exploration into Wartime Ciphers - Enigma and Lorenz '/>

Playing Soldiers: An Exploration into Wartime Ciphers - Enigma and Lorenz

I had a bit of time over the Summer, and thought I would do something that I have been meaning to do for a long time. Create a software implementation of an Enigma and a Lorenz machine. While I was at it, I thought that I might as well take a look at how the machines could be broken.

For the programs themselves, head over to the project page. The rest of this post is an explanation of how the machines work and the low-hanging fruit when it comes to vulnerabilities.

Preface

Enigma and Lorenz are two series of electromechanical cipher machines that saw extensive use by the Axis powers during World War II. Enigma was primarily used by the German Navy, Air Force, and Army as a method to encrypt communications. Lorenz machines were used from 1940 onwards and were the preferred encryption method for communications between German High Command. The Enigma and Lorenz machines formed the vast majority of traffic decrypted at the British Government Code and Ciphers School (GC&CS) in Bletchley Park.

Enigma

Enigma was the most widely used of the two machines, and the largest share traffic flowing through Bletchley Park was encrypted by Enigma. With over 40,000 machines constructed (compared to the approximately 200 Lorenz machines), Enigma was the priority of Allied codebreakers, especially during the early stages of the war.

Workings

German military Enigma machines are made up of three distinct parts: the rotors, the reflector and the plugboard.

The Plugboard

The plugboard is the simplest part of the encryption process but is what separates military Enigma from the commercial variant used before the war. The plugboard is a grid of contacts with each contact relating to a letter. Two ends of a plug would be inserted into two of the letters to electronically connect them. The connections on the plugboard would cause letters to be swapped before and after encryption. Each Enigma machine came with 13 plugs, with only up to 10 ever used at one time.

The Rotors

The rotors are the most distinctive element of the Enigma machine. All German military Enigma machines featured three conventional rotors. A later Enigma model, the M4, is often referred to as having four rotors however this ‘fourth’ rotor behaves more like a reflector than a conventional rotor.

A rotor is very straightforward electronically: it maps a single input on one side, to an output on another side, through a wire. There were 26 mappings on each rotor, one for each letter of the alphabet, and each mapping was marked with either a number from 1 to 26 or the letters A to Z. The topmost marking could be seen through a small window on the machine, and this was used for setting an initial configuration. An Enigma rotor also has a “ring setting”. The markings on the rotor can be twisted independent of the rotor itself and so the mapping represented by “A” on ring setting 0 would be represented by a “Z” on ring setting 1. It effectively offsets the outer casing and notch from the inner wiring. The ring setting is set at the beginning of the message and remains the same throughout.

On each keypress the rightmost rotor is rotated towards the operator by a single step. Each rotor also has one or two small notches aligned with one or two of the steps. Usually, when the right rotor turns it blocks the other two rotors from spinning. However, if the rotor is in a position where the notch aligns with the mechanism, the rotor to the left of it can rotate one step. The same is true for the middle and left rotors. The effect is similar to something like the odometer on a car: at a certain point in the rotation of the right rotor, the middle rotor is increased by one, and at a certain point in the rotation of the middle rotor, the left rotor is increased by one.

Note on Double Stepping

There is a quirk with the mechanism of German Enigma machines which means the middle rotor will also rotate if it is at its notch position even if the rotor to the right of it is not.

The Reflector

The reflector is electronically similar to a rotor in that it has a series of input and output contacts that are mapped with wiring, however there is no rotation of a reflector during transmission of the message. A reflector is set at the beginning of a message and remains static throughout. The ‘fourth’ rotor of Enigma M4 also remains static throughout the message and therefore it somewhat of a misnomer to call it a ‘fourth’ rotor. In fact, using the ‘beta’ wheel as the fourth rotor and the UKW-b wheel as the reflector in M4 Enigma is logically identical to the UKW-B reflector of three-wheel Enigma.

Stages of Encryption

To fully understand the stages of Enigma encryption, it is best to show a worked example. In this example we are going to encrypt the character ‘A’ with rotors III, II, and I in default position, with A mapped to T on the plugboard, and with the ring setting of rotor I set to 2. To run this configuration in the provided simulator the command would be:

enigma -m "A" -l "III 1 0" -c "II 1 0" -r "I 1 2" -ukw "B" -plugs "A:T"

Character 'A' Pressed

When character A is pressed, the right most rotor rotates forward by one. The right rotor was not at its notch position, so it is the only rotor that moves. An electrical signal is then sent through the machine.

The first stage is the plugboard. Our plugboard maps A to a T so our electrical signal comes out of contact T of the plugboard.
Because rotor I has been shifted by one, output T of the plugboard actually goes into input U of rotor I. The input contact U maps to the output contact U and so this receives the electrical signal.
Rotor I’s contact U is connected to Rotor II’s input contact T so this is the input point of the rotor. T is wired to N in this rotor, so a signal is outputted on this contact.
Rotor II’s contact N is connected to Rotor III’s input contact N so this receives the signal. Input N is wired to output N so electricity is passed out at this contact.
Output N goes into the reflector’s input N, travels to the mapped output point K and then travels back through all previous 3 rotor along whichever wirings are available in that direction.
K is mapped to U so Rotor III feeds electricity to the contact U on Rotor II’s left hand side.
U is mapped to H so Rotor II feeds electricity to its output contact H.
Because Rotor I is rotated by 1 step, Rotor II’s right contact H is touching Rotor I’s left contact I so this is the input point. I maps to H so electricity is fed to this output contact.
Once again, because I is rotated by I, contact H is actually in connected to the G contact on the plugboard. Because we have no mapping for G, it passes through unchanged, and the G lamp lights ups. G becomes our ciphertext.

Methods of Decryption

Brute Force

The most naïve of decryption methods would be to brute force the initial settings. This would be incredibly difficult, even on a modern machine, due to the large number of possibilities:

Possible Rotor Combinations = 5×4×3=60 (Picking 3 rotors from 5)

Possible Rotor Settings = 26×26×26=17,576 (26 settings for each of the 3 rotors)

Possible Plugboard Settings = 26!/(6!10!2^10)=150,738,274,937,250 (26 letters, order of 10 pairs does not matter, 6 letters go unused, mappings are bi-directional)

Total Combinations = 158,962,555,217,826,360,000

This is only using a three rotor Enigma machine where three rotors are chosen from five. The Navy could choose three rotors from eight, and the M4 Enigma had even more combinations with its two reflectors.

Brute forcing Enigma was impossible at the time and is still infeasible as a method of decryption on modern machines in a reasonable period.

Analytical Methods

Because of the infeasibility of brute force methods, cryptanalysis had to be performed to reduce the search space to something reasonable.

Known-plaintext Attacks

One of the key flaws in Enigma’s encryption process was that, due to the nature of the reflector, a letter in plaintext could never map to the same letter in the ciphertext. The primary method of utilising this flaw came in using either known plaintext or guessed plaintext. If a message was sent with partially known content in a guessable format, then you could use the flaw to work out where certain words may appear in the ciphertext and then reverse engineer the settings from that.

An example of this might be a weather report. A weather report is likely to contain the German word for ‘weather report’, ‘wetterbericht’. You could see where ‘wetterbericht’ would plausibly fit in the message (no plaintext letters match ciphertext letters) and only go through combinations where that outcome is possible. A piece of known plaintext such as this was known as a ‘crib’.

Cribs could also be used to identify plugboard combinations. By going through the crib, we could make certain assumptions and develop trees of possible plugboard settings. If one of our assumptions is contradicted along the tree, then we can rule out that whole branch of plugboard settings without ever needing to try them. Techniques like plugboard checking can help to reduce the search space significantly. The search space became small enough that even in the 40s it was possible to use these techniques to reduce the problem electromechanically using devices known as Bombes.

Ciphertext-only Attack

All successful cryptanalysis on Enigma during the war focused on known-plaintext attack methods. Modern computing power combined with statistical methods allows for certain plaintext-only methods of decryption. On such plaintext-only attack involves calculating the index of coincidence. Index of coincidence (IoC) is a measure of how likely to it is to draw two letters from a piece of text, and for those two letters to be the same. Index of coincidence is useful in decrypting because the distribution of letters in a language is different to that of random characters (A piece of random text has an average IoC of 1.0 whereas German text has an average IoC of 2.05). Enigma is vulnerable to the use of IoC as a method of decryption since if one of either the rotors, ring settings, or plugboard are correct, the message will be slightly closer to the plaintext and therefore have a slightly higher IoC.

If you were to run all possible rotor combinations with default ring settings, no plugboard, and the message was sufficiently long, the IoC of the correct rotor combination would be slightly higher than random. Being able to solve, or at least narrow, the problem in individual elements removes the multiplicative element of brute force and makes the problem more computationally feasible. Likewise, having one of the plugboard settings correct will produce a better IoC than having none of them correct.

IoC is only useful when you have enough text to become statistically significant, with Enigma this would be somewhere between 600-1500 characters depending on plugboard settings. The German military purposely kept Enigma messages short for this very reason, as IoC was known at the time. There were, however, occasions where operational mistakes or multiple messages using the same settings would result in long strings of characters enciphered the same.

It is likely that IoC could be used in conjunction with known-plaintext attacks to create a much more efficient method of decryption since both use alternative ways to narrow the search-space.

Lorenz

As previously mentioned, The Lorenz machine was used to encipher the most secret intelligence between Hitler and his generals. The nature of the traffic meant that the cryptanalysts at Bletchley Park had much fewer messages to try and decrypt, and the lack of machines meant that it was less likely for them to get a physical machine to test with. In fact, almost all cryptanalysis on Lorenz was done without any access to the physical hardware as the first machine was only captured by the allies late in the war as Europe was being liberated. However, messages did tend to be longer due to the fact the Lorenz machine was much easier to use.

Workings

Unlike Enigma, which was sent by morse, the Lorenz machine used teleprinter code which was automatically ciphered and deciphered. An operator would type their message in plaintext and the receiver would see the incoming message in plaintext with decryption only occurring for transmission.

Lorenz signals used, what would now be referred to as, a 5-bit encoding scheme known as ITA2. Plaintext characters would be represented as 5 dots or spaces on 5-hole paper tape, and this is what would be enciphered. The encipherment is logically straightforward, the plaintext ITA2 stream would have a bitwise XOR operation performed on it with a pseudorandom keystream. It is the cracking of how the keystream was generated that would be vital in uncovering the message.

For UX reasons, the ITA scheme used by the Lorenz simulator is slightly different to the standard ITA2 schema. The simulator uses the following mapping:

Binary Code	Letter Shift	Figure Shift
`00000`	±	±
`00001`	T	5
`00010`	_	_
`00011`	O	9
`00100`	SPACE	!
`00101`	H	£
`00110`	N	,
`00111`	M	.
`01000`	\|	\|
`01001`	L	)
`01010`	R	4
`01011`	G	&
`01100`	I	8
`01101`	P	0
`01110`	C	:
`01111`	V	=
`10000`	E	3
`10001`	Z	+
`10010`	D	#
`10011`	B	?
`10100`	S	\
`10101`	Y	6
`10110`	F	%
`10111`	X	/
`11000`	A	-
`11001`	W	2
`11010`	J	$
`11011`	^ (Figure Shift)	^ (Figure Shift)
`11100`	U	7
`11101`	Q	1
`11110`	K	(
`11111`	* (Letter Shift)	* (Letter Shift)

Keystream Generation

Like Enigma, the Lorenz machine used rotors, however there were 12 rather than the 3 found in Enigma. The rotors can be divided into three sets: the ψ wheels, the χ wheels, and the μ wheels.

Each rotor has a number of pins across its circumference. The number of pins on a rotor is co-prime with the number of pins on the other rotors, giving a very long period before a key sequence is repeated. The pins on the rotor can be set to either on or off (equivalent to 1 or 0 on a bit pattern).

When a key is pressed, the 5-bit input is XORed with the bit pattern across the χ wheels, then again with the bit pattern across the ψ wheels. The wheels are then stepped depending on the position of the other wheels.

Rotor Stepping

After the XORing, all χ rotors step forward by one.
The ψ wheels only step forward by one if μ2 is in a ‘on’ positions
Then, μ1 will always step forward by one
If μ1 is in an ‘on’ position then μ2 will also step forward by one.

Later models of the Lorenz machine had further limitations placed on when a rotor would step forward.

Methods of Decryption

As with Enigma, Lorenz machines were far too complex to have any hope of using brute force techniques to acquire the initial settings.

Depths

As with Enigma, having credible guesses of what parts of the plaintext might be can go a long way into decrypting the whole message. If two messages share the same key then it becomes a much easier task.

Consider the following, where C1 and C2 are two cipher texts, P1 and P2 are two plaintexts, and K is a key they have in common.

$$C_1=P_1\oplus K$$ $$C_2=P_2\oplus K$$ $$C_1\oplus C_2=P_1{\oplus P}_2\oplus K\oplus K$$ $$\mathrm{Since\ }K\oplus K=0:$$ $$C_1\oplus C_2=P_1{\oplus P}_2$$ $$\mathrm{let\ }F=C_1\oplus C_2$$

As can be seen, by having two messages that use the same key, you can achieve a combination (F) of the two plaintexts by XORing the two ciphertexts. The resulting amalgamation contains no part of the keystream.

The result of this is, if we can guess a part of one plaintext, and XOR it with F, then we will get a part of the other piece of plaintext. One powerful instance of this weakness proving useful came when a German officer sent two long messages using the same key with slightly different content. First a message was sent, then the receiver requested the message be sent again due to a failure on their end, the sender sent the message again but, because they were more impatient the second time around, started using abbreviations in their message.

Worked Example

Say we have the message:

Plaintext:      REGIMENT NUMBER 5 WILL ATTACK WEST POINT AT DAWN
Ciphertext:     Z_ZFINNOV_YQM*PL*_JL^6?-1!::9=|+,(2(:1.$,1-1*OKVIL

And then our operator comes back to us and asks us to repeat the message. We might send back:

Plaintext:      REGIMENT NO 5 WILL ATK W POINT AT DAWN
Ciphertext:     Z_ZFINNOV_RKVCALQVI NT^=1£&_-03'^%8^-|13

If we XOR our two cipher text messages we get:

F:              ±±±±±±±±±±*O|ZY±_PFPQS|X±THI^_-£1|6£%6$4

In this particular ITA alphabet, ± is the null character and so we know that the characters in those positions are the same in both messages.

Let us suppose that we know that a message is going to start with a regiment number, but we don’t know where they will attack. We can XOR our crib with F to see if we can get some of the other message:

F:              ±±±±±±±±±±*O|ZY±_PFPQS|X±THI^_-£1|6£%6$4
Crib:           REGIMENT NUMBER
F XOR CRIB:     REGIMENT NO ^5*

We can see that be XORing our crib against the message, we get the contents of the other message. Since they are messages with the same intent, we can take a guess that our first message will also refer to regiment 5, this can form the basis of a new crib:

F:              ±±±±±±±±±±*O|ZY±_PFPQS|X±THI^_-£1|6£%6$4
Crib:           REGIMENT NUMBER ^5*
F XOR Crib:     REGIMENT NO ^5* WI

Here we have gathered two more letters. We can keep feeding cribs into each other to read out the entire message:

F:              ±±±±±±±±±±*O|ZY±_PFPQS|X±THI^_-£1|6£%6$4
Crib:           REGIMENT NUMBER ^5* WI
F XOR Crib:     REGIMENT NO ^5* WILL A
Crib:           REGIMENT NUMBER ^5* WILL A
F XOR Crib:     R REGIMENT NO ^5* WILL ATK W

At points like this we can see another likely deviation in the messages. We have found out that the second message uses the phrase ATK, one assumption we could make is that they used the full word ATTACK in the original message.

F:              ±±±±±±±±±±*O|ZY±_PFPQS|X±THI^_-£1|6£%6$4
Crib:           REGIMENT NUMBER ^5* WILL ATTACK
F XOR Crib:     REGIMENT NO ^5* WILL ATK W POIN

Likewise, a cryptanalyst might also assume that W could mean WEST. Realistically cryptanalysts would try various combinations to see which one creates reasonable plaintext.

F:              ±±±±±±±±±±*O|ZY±_PFPQS|X±THI^_-£1|6£%6$4
Crib:           REGIMENT NUMBER ^5* WILL ATTACK WEST POI
F XOR Crib:     REGIMENT NO ^5* WILL ATK W POINT AT DAWN

We now have both messages in the clear without ever needing to know what the settings were of the Lorenz machine or what the key stream was. The only outright guess we needed to make was that the message would start with the phrase “REGIMENT NUMBER”, and from that we found out which regiment it was, what they were doing, and when they were doing it.

This was not the only technique used in the breaking of Lorenz traffic, but it is one of the most potent. Such an attack also highlights how operational failures can severely jeopardise the use of a system that is quite secure from a cryptographic standpoint.

Bibliography

Computerphile. (2014, December 9). Tackling Enigma (Turing's Enigma Problem Part 2) - Computerphile. Retrieved from YouTube: https://www.youtube.com/watch?v=kj_7Jc1mS9k

Computerphile. (2014, November 28). Turing's Enigma Problem (Part 1) - Computerphile. Retrieved from YouTube: https://www.youtube.com/watch?v=d2NWPG2gB_A

Computerphile. (2015, July 1). Fishy Codes: Bletchley's Other Secret - Computerphile. Retrieved from YouTube: https://youtu.be/Ou_9ntYRzzw

Computerphile. (2015, September 16). Zig Zag Decryption - Computerphile. Retrieved from YouTube: https://www.youtube.com/watch?v=yxx3Bkmv3ck

Computerphile. (2018, September 5). Exploiting the Tiltman Break - Computerphile. Retrieved from YouTube: https://www.youtube.com/watch?v=-hrNWRtDr7Y

Computerphile. (2021, April 21). Cracking Enigma in 2021 - Computerphile. Retrieved from YouTube: https://www.youtube.com/watch?v=RzWB5jL5RX0

Good, J., Michie, D., & Timms, G. (1945). General Report on Tunny.

Jackson, J. (2022, August 15). The National Museum of Computing. Retrieved from The Enigma Machine: https://www.tnmoc.org/bh-2-the-enigma-machine

Numberphile. (2013, January 10). 158,962,555,217,826,360,000 (Enigma Machine) - Numberphile. Retrieved from YouTube: https://www.youtube.com/watch?v=G2_Q9FoD-oQ

Numberphile. (2013, January 14). Flaw in the Enigma Code - Numberphile. Retrieved from YouTube: https://www.youtube.com/watch?v=V4V2bpZlqx8

Owen, J. (2021, December 11). How did the Enigma machine work? Retrieved from YouTube: https://www.youtube.com/watch?v=ybkkiGtJmkM

Stichting Cryptomuseum. (2022, August 12). Retrieved from Crypto Museum: https://www.cryptomuseum.com/index.htm

Thaophialuang, L. (2019, January 24). Enigma Simulator. Retrieved from Enigma Cipher: https://piotte13.github.io/enigma-cipher/

For Fonts in My Life

Sat, 11 Mar 2023 16:39:17 +0000

For Fonts in My Life '/>

For Fonts in My Life

Header image by Andrew Wright

It is dissertation writing season at University, and this means one thing, document creation. There are a few schools of thought when it comes to document creation.

The Office Clerk

A person who will use 12pt Calibri, have headings in 'Blue, Accent 1', and submit it as a .docx file if they can. Likely records videos in portrait.

The WYSIWYG Wizards

Someone who still uses Microsoft Office but will change font, margins, bullet styles, and may even go as far as designing a custom template.

The Typesetters

Someone who uses dedicated document preparation software. Will spend hours searching for the correct typeface. They spend more time on formatting than on writing. Do not mention Arial in the company of this person.

You will fall somewhere along this spectrum. Most likely somewhere between the first and second. I am here as a counterbalance. To show you why you should care about your documents. But mainly I'm here to gush about typefaces.

Warning, unqualified pretentious font talk ahead!

Words Count

A good typeface can elevate a document to something more than the sum of its parts. A typeface has one purpose, to present text on a page for it to be read. But it is the way in which it achieves this purpose where there is space. Space for creativity, for expression, for sentiment. When I make a document that I want to present, I want that document to say something about me.

The written word is a decidedly constrictive medium for visual creativity. Especially when those words are part of a dull report, or distinctly verbose thesis. The writing itself is the star of the show of course, but that show should be presented in as grand a theatre as possible.

Here I have the same line in the two stalwarts of Microsoft Office, Times New Roman and Calibri. These typefaces lay at opposite ends of design language. One is a traditional Roman style serif designed for a newspaper in the '30s, but inspired by the likes of Baskerville from the 18th century. The other is a humanist sans-serif, designed for digital displays, with edges smooth enough to leave near small children. Each of these have their place (probably), but it is inarguable that Times gives a more powerful presentation of the words of INGSOC.

This is a rather extreme example. Typefaces, like any art, mostly boil down to taste and personal preference. As much as I dislike Calibri, I respect someone's choice to use it. The problem is, people very rarely choose it. They use it because they didn't make a choice, and apathy is the fashion of the meek. If there is one thing I want you to get from this article, it is to make a choice. Let every part of your document's aesthetic be a conscious decision, own your typography.

IBM Plex Sans

As an example of a conscious font choice, I present my use of IBM Plex Sans in bold as something I commonly use as a heading font.

I adore the era of computing when teletypes reigned supreme and graphics were optional. I wouldn't like to live then, you understand. Much in the same way that people idolise the wild west, when in reality it was organised crime mixed with dysentery. I just like the smell I get in my ears when I look at the photographs.

Four of the biggest players in computing during this era were IBM, DEC, Bell Labs (owned by AT&T), and General Electric. IBM were the largest, they developed the first mainframe computers that could be sold in large (for the time) numbers.

Bell Labs were the research powerhouse of the industry. The shear importance of Bell Labs in the history of science and computing cannot be overstated. The transistor, the cosmic microwave background radiation, the laser, the CCD, the concept of bits, C, C++, Unix, nine Nobel Prizes, five turing awards... The list of achievements is astounding and is worthy of several documentaries.

The GE computing division (later sold to Honeywell) and DEC were not as large as IBM but still formed two of DeBUNCH of IBM competitors.

By 1988, the 5 of them would represent the 4th, 6th, 8th, 28th, and 38th largest companies in America. They were by far the largest technology companies of their day and represented the dawn of the information age.

This duality of being the cutting edge modernity and being multinational corporations came through in their typography. Above are excerpts from the Bell System Technical Journal of December 1979, the IBM System/360 user manual, the GE-645 system manual, and the PDP-11 Processor handbook.

Bell and DEC are using the Neo-Grotesque Helvetica, with IBM and G.E using the Geometric Futura.

Helvetica

Helvetica is a typeface that is both lauded and besmirched. Its design is heralded, but it's use as the symbol of modern corporate America meant it hit a level of saturation that has left a sour taste in the mouth of those from a prior generation.

And they are perfectly correct of course. Helvetica found an audience in the large companies of the 80s and 90s. It is non-confrontational, it has no outlandish flair. And the absence of serifs along with its Swiss modernism moves it away from the archaic Garamonds and Caslons of the world. But I think that its use amongst the billion-dollar conglomerates of yesteryear should not be a reason to dismiss this typeface.

For a start, the saturation of Helvetica means that brands have began to move away from it. American Airlines changed its font in 2013. And today's major companies (Google, Tesla, Microsoft) no longer see it as representing modernity. So perhaps it's reasonable to look at why Helvetica was used in the first place.

The strength of helvetica is in the way it can hold its own on a page.

If we compare it against the Humanist Gill Sans, I just feel it sits so much better. It brings an air of German functionalist efficiency that Gill Sans doesn't.

The stand-out character in Helvetica is the capital R. Helvetica feels placed, in position, structurally sound. One of the reasons for this is the way all of its terminals are either perfectly vertical or horizontal. Look at the way the C opposes itself, the tail of the y facing left, and the s finishing straight up and straight down. Helvetica locks itself into place with these, it grabs the page.

On the other hand, Gill Sans, and a lot of humanist typefaces, feel like they're ever so slightly about to fall over. They seem to me like they always need one more letter next to it just to support the previous one. Of course, this run-on nature makes them wonderful for body text. They allow text to flow easily, whereas the rigid nature of helvetica is slightly too uniform to make it nice to read in large chunks. For header text though, Helvetica can reign supreme.

Futura

Speaking of Supreme, Futura. It is what is known as a geometric typeface, it's characters are formed from circles and straight lines. It's foundations in geometry make it a perennial staple in anything wanting to convey as modern or futuristic. Wes Anderson, Stanley Kubrick, and NASA have all made extensive use of its sharp angles and perfect curves.

It's easy to see why it is so popular. It just screams technology. It's abstract characters redefine type in terms of mathematics and the result is something notably artificial.

Man and Machine

IBM Plex falls into the category of Grotesque fonts. It shares quite a few similarities with Helvetica and its predecessor Akzidenz-Grotesque. The main similarity it has with Akzidenz and Futura over Helvetica is its more gothic R with a diagonal leg.

Although it does not share any letter forms with Futura, IBM plex still has a distinct geometric machined quality. Its counters are not perfect circles, but they're not the ovals of more Humanist or Grotesque typefaces either. Characters like C, G, O, and Q all have a stadium shaped counter. Not an elemental shape but still with foundations in geometry. The central "pipe" like leg also has an artificial quality but once again has its roots in machinery rather than formal design. The only real Humanist design choice is the double-storey g. A strange decision thematically, but I think it's little pipe ear is quite charming.

IBM have created a typeface that harks back to the pre-digital age. Just like IBM, the grotesque typeface was born out of the Industrial era and found its greatest success between the 50s and 90s (with Helvetica). It's machined nature doesn't distract either. It manages to be contemporary without being sleek.

Charter

Into the realm of serifs, more specifically serif body typefaces. I'm more of a fan of using a serif for the main text of a document. Some people say it is more readable, I just feel it fits with my more classic, timeless aesthetic.

Serifs can get a bit tricky to get right because it depends on the context. Something like Baskerville looks quite beautiful in print. Its serifs taper thin, and it has a very high stroke contrast that shines on the page of a book. To see what I mean, look at the capital A. The left hand side is much thinner than the right.

The problem with these older typefaces, designed to look great on paper, is they don't transfer to the digital screen very well. At the lower detail densities of screens, these thin strokes can become hard to read and harm legibility. You will notice that modern serif typefaces designed for screens or home laser printers will have much lower stroke contrast. They also tend to have a slightly higher x-height.

The obvious pairing for IBM Plex Sans should be IBM Plex Serif, right? Well it left me feeling a bit cold. I can imagine using it for UI, but overall I can't gel with its modernism. Its Bodoni-style sharp, rectangle serifs feel a bit too geometric for me. It also has a reasonably tall x-height, moving away from the traditional look that I want.

Charter strikes a good middle-ground. Its x-height is a bit shorter, it's serifs tapered (but not excessively), and a slightly thinner average stroke width from something like Georgia (Also by Matthew Carter).

Of course, I'll chop and change things depending on what a document should look like, but I hope I've given you enough of an introduction into typefaces that you may dip your toe into something other than the default. If you want more guidance on things other than typography then I can't recommend Butterick's Practical Typography enough.

And just for the sake of it, here are some more honorable font mentions.

Serif

Sans-serif

Luke Briggs

The First Line of Code

The First Line of Code

A Game of Two Halves

A Game of Two Halves

Inspection & Dissection: Ping! – My First Game in Unity

Inspection & Dissection: Ping! – My First Game in Unity

Where I Began

Where I Began

Making a Mobile App – Chapter 1: A Few False Starts

Making a Mobile App – Chapter 1: A Few False Starts

Making a Mobile App – Chapter 2: A Bad Worker Blames His Tools

Making a Mobile App – Chapter 2: A Bad Worker Blames His Tools

Making a Mobile App – Chapter 3: The Setup

Making a Mobile App – Chapter 3: The Setup

Making a Mobile App – Chapter 4: Fast and Flash

Making a Mobile App – Chapter 4: Fast and Flash

Branding

Purpose

Foolproofness and Expandability

Making a Mobile App – Chapter 5: Go Your Own Way

Making a Mobile App – Chapter 5: Go Your Own Way

Probably

Probably

Inspection & Dissection: Dice-Jack

Inspection & Dissection: Dice-Jack

Behind the prototype

The Evolution

The Layperson’s Guide to LaTeX

The Layperson’s Guide to LaTeX

What is this LaTeX and Why is it Not Pronounced Like That?

Isn’t This a Bit of a Faff?

How Good Does it Look?

Where Do I Begin?

Further Notes

Inspection & Dissection: This Site!

Inspection & Dissection: This Site!

Why have a website?

Why make your own website?

Where did you start?

What extra things have you done?

Goodbye Windows, I Hardly GNU ya

Goodbye Windows, I Hardly GNU ya

The Linux way of thinking

Technical Details

Why Manjaro?

You Can Now Follow Me on RSS!

You Can Now Follow Me on RSS!

Inspection and Dissection: Type or Die

Inspection and Dissection: Type or Die

Where Hugo I Go

Where Hugo I Go

Technology

New Logo

Flatpak: Instructions Not Included

Flatpak: Instructions Not Included

Context

Python Deployment

Linux Binary Woes

Build Systems

Flatpak

The Trouble with Python, Qt and Flatpak

Lasers to the rescue

Further Reading

Inspection and Dissection: Pepys - A Straightforward Markdown Editor

Inspection and Dissection: Pepys - A Straightforward Markdown Editor

Brief Description

Background

A Rocky Start

A Walk in the Dark

Syntax Highlighting

HTML Preview

The Killer Feature

Calendar File Selector

Release

A Retrospective

Special Thanks

Full Release: newcastle-bst - Harvard referencing style as recommended by Newcastle University

Full Release: newcastle-bst - Harvard referencing style as recommended by Newcastle University

Release Notes