The Sixth Term Examination Paper (STEP) is a test administered by the University of Cambridge which is required for admission to their mathematics courses. It consists of 2 papers, STEP II and STEP III (STEP I was discontinued in 2020), each 3 hours long, consisting of several questions from pure mathematics, mechanics and statistics & probability, covering nothing more than the A-Level Mathematics and Further Mathematics syllabus (and yes, that means that for an IB student like me, I have a lot of learning to do, like matrices and literally all of mechanics).
Only 6 questions, 20 marks each, are assessed for a maximum score of 120 marks on each paper. They are then converted to a grade ranging from S, 1, 2, 3, U. A grade of 1, 1 on the STEP II/III is typically required. On average, a score of 69/120 on the STEP II and 62 on the STEP III will get you a 1, 1. For an S, S, you need a score of approximately 92/120 and 84 on the STEP II and III, respectively.
Yes, if you couldn't tell already, the STEP is rather notorious for its merciless difficulty. For this reason, it is taken in June of year 13/grade 12 after you have finished your school work (including the IB exams).
If you want to take a shot at some STEP questions, visit the STEP database (which is also the database that this STEP bot uses).
The STEP discord bot which I created has two key functions:
Visit the GitHub page to see the code and find out more about how this bot works.
I used discord.py rewrite and structured my code using a Cog, which are basically classes to group your Discord functions together. I heavily relied on discord.ext.commands, which allows you to easily create commands without the hassle of a more traditional approach which would require you to manually filter the command input.
Usually, for a web scraping task, I would use requests and BeautifulSoup, but this one was special. Firstly, the STEP database API had a direct link to each of the questions as a png file, like this: f"https://stepdatabase.maths.org/database/db/{_year}/{_year}-S{paper}-Q{question}.png". This meant that BeautifulSoup was not necessary, and I could simply access the URL. Furthermore, since a Discord bot is created using asynchronous programming, instead of using requests which is a synchronous library, I used aiohttp instead.
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
if response.status != 200:
return await ctx.channel.send('Could not find question...')
data = BytesIO(await response.read())
k = await ctx.send(file=discord.File(data, filename=f"SPOILER_{year}-S{paper}-Q{question}.png"))
await k.add_reaction("\u2705")
await k.add_reaction("\u274C")
What this code block does is the following:
ctx.send() where ctx is called the context in which the command was executedAnd that's it! It's really that simple. Now on to the hard part...
In order to save the question completion status of all of my users, I needed to create a database. Whilst Sqlite (using Python's sqlite3 library) is my top choice, since I wanted to deploy it to Heroku which has an ephemeral filesystem (meaning that files written to disk will disappear after 24h, as opposed to a persistent storage filesystem wherein files will exist forever), I needed to use PostgreSQL instead.
Firstly, however, I would like to share some sources that were extremely helpful when I was setting up and using PostgreSQL myself. Unfortunately, this is not an exhaustive list as I consulted many other sources, and have lost them in the process. I hope some of these will help you on your own journey to using PostgreSQL.
psqlpsql on Terminal to display tablesWith that out of the way, the first step of using PostgreSQL was probably the biggest challenge of this entire project. Yes, good start, I know. But installing psycopg2, the library required to access PostgreSQL, is troublesome for many people, and I was no exception. Just search up "install psycopg2 error" and see for yourself the sheer number of people with this problem!
Since I tried so many things (such as installing xcode-select, setting some environment variables in the .zshrc file in my home directory, installing some other things with homebrew, etc) that I do not know what "fixes" contributed to my success, one thing I can say for certain is that installing psycopg2==2.8.5 as opposed to 2.8.6 (the latest version) did the trick for me. I have no idea why, but as soon as I told pip to install 2.8.5, it just magically worked.
Next, I then installed Postgres and with it, PgAdmin 4, on to my computer. This was extremely simple thanks to the guide I posted above (first link you see). The only thing to note is that your master password is separate from your user password. The master password is used to log in to PgAdmin 4 itself, whereas your user password is used to log in to a specific user on both PgAdmin 4 and Terminal (via psql). This is necessary because you could have multiple users with different databases.
I then accessed my database and created a table using the last link in the list above.
conn = psycopg2.connect(host=DB_HOST, database=DB_NAME, user=DB_USER, password=DB_PASS)
with conn:
with conn.cursor() as cur:
cur.execute('''
CREATE TABLE IF NOT EXISTS "members"(
"id" BIGSERIAL,
"completed" TEXT,
"username" VARCHAR(100)
)''')
I needed a BIGSERIAL as opposed to a SERIAL since Discord user IDs are quite big numbers. The completed column stores all the questions completed by the user as a string.
At this point, I tested PgAdmin 4 and psql on Terminal to view the database. It was working perfectly. So next, I executed some SQL commands using psycopg2:
with conn: #automatically conn.commit()
with conn.cursor() as cur: #automatically cur.close()
cur.execute('''SELECT * FROM members WHERE id = %s''', (ctx.author.id, )) # get the users with matching id (should be one)
res = cur.fetchone()
#Add/update users
if res == None: # user is not in db
#id, completed (should be empty), username
cur.execute("INSERT INTO members VALUES (%s,%s,%s)", (int(ctx.author.id), "", f"{ctx.author.name}#{ctx.author.discriminator}"))
print(f"User {ctx.author.name}#{ctx.author.discriminator}, {ctx.author.id} added to database")
else: #user is in db, they may have changed username so update
cur.execute("UPDATE members SET username = %s WHERE id = %s", (f"{ctx.author.name}#{ctx.author.discriminator}", ctx.author.id))
#Check question completion
cur.execute("SELECT completed FROM members WHERE id = %s", (ctx.author.id,))
completed_str = cur.fetchone()[0]
emoji = "\u2705" if f"{year} {paper} {question}" in completed_str else "\u274C"
edit = await ctx.send(f"Completed? {emoji}")
self.current.append(Listener(k, ctx.author, ctx.message, edit, f"{year} {paper} {question}"))
Here, I first checked if the user who executed the command was already in the database. Then, I checked to see if that question was completed by that user, and sent an appropriate emoji to indicate the completion.
The last line of code is the tricky part. Listener is a class that is responsible for holding the state of this particular instance of a command execution.
class Listener:
def __init__(self, k, author, message, edit, question):
self.k = k
self.author = author
self.message = message
self.edit_message = edit #Completed?
self.question = question #year S_paper question
self.time = time.time()
class Step(commands.Cog):
def __init__(self, bot):
self.bot = bot
self.current = []
async def check_listeners(self): #clear reactions after 3600 seconds
i = 0
for start_time in [listener.time for listener in self.current]:
if time.time() - start_time > 3600:
i += 1
for listener in self.current[:i]:
await listener.k.clear_reactions()
self.current = self.current[i:]
@commands.Cog.listener()
async def on_ready(self):
print(f"Logged in as {self.bot}")
while True:
await self.check_listeners()
await asyncio.sleep(10)
This is how it works:
Listener and appended to a list called self.currentasyncio.sleep() instead of time.sleep(), since we want asynchronous programming!), the check_listeners() function is called.This is necessary to make sure the bot is not flooded with tons of data that is never cleaned. After that, I wrote a command that handles a reaction by a user.
@commands.Cog.listener()
async def on_reaction_add(self, reaction, user):
for listener in self.current:
if user.id != bot.user.id:
ctx = ReactionContext(user, listener.message.channel)
if reaction.emoji == "❌" and reaction.message.id == listener.k.id:
await listener.k.remove_reaction("❌", user)
await self.uncomplete(ctx, *listener.question.split(" "))
#only if the user is correct, then edit the Correct? message
if user.id == listener.author.id:
new_content = listener.edit_message.content.replace("✅", "❌")
await listener.edit_message.edit(content=new_content)
elif reaction.emoji == "✅" and reaction.message.id == listener.k.id:
await listener.k.remove_reaction("✅", user)
await self.complete(ctx, *listener.question.split(" "))
if user.id == listener.author.id:
new_content = listener.edit_message.content.replace("❌", "✅")
await listener.edit_message.edit(content=new_content)
This is pretty self explanatory: it goes through all the commands stored in the current list, and calls the appropriate functions. A trick here is the ReactionContext class which is created to emulate Discord's context class which I mentioned previously.
@commands.command(aliases=["c"])
async def complete(self, ctx, *args):
# year, paper, question = process_input(args)
if k := process_input(args):
year, paper, question = k
else:
return await ctx.channel.send("Invalid arguments")
with conn:
with conn.cursor() as cur:
cur.execute("SELECT completed FROM members WHERE id = %s", (ctx.author.id, ))
a = cur.fetchone()
if f"{year} {paper} {question}" not in a[0]: #if user has not already completed the paper
cur.execute("UPDATE members SET completed = %s WHERE id = %s", (a[0]+f"{year} {paper} {question}", ctx.author.id))
await ctx.send(f"{year} S{paper} Q{question} has been marked as complete for {ctx.author.name}.")
else:
await ctx.send(f"{year} S{paper} Q{question} is already complete for {ctx.author.name}.")
Finally, here is the function responsible for handling a question being marked as complete (the incomplete function is very similar). It first processes the input using regex (I really wanted to use Python 3.9's new str.removeprefix() feature, but unfortunately this wasn't exactly what I needed) and handles it using the much beloved walrus operator (introduced in Python 3.8). It then connects to postgres again and adds the question to the completed column if it was not already completed.
Unlike my other web projects which required a complicated setup with Flask commands to create/setup the database and the admin users and Flask-Migrate to migrate the database (a post on this will be coming soon!), deploying this bot was rather simple.
After pushing to Git and creating an app on Heroku as usual, I attached a PostgreSQL database like I usually do for my web projects. The only problem was that I did not know what values to put for the database host, name, username and password. A quick google search led me to this question on Stack Overflow.
import urllib.parse as urlparse url = urlparse.urlparse(os.environ['DATABASE_URL']) dbname = url.path[1:] user = url.username password = url.password host = url.hostname port = url.port
This is the piece of code that is essential for using deploying a PostgreSQL database using psycopg2 to Heroku (credits to user @J0ANMM for this). On your local computer, the database is something you created yourself, so an example configuration might be something like this:
DB_HOST = localhost DB_NAME = step-bot DB_USER = postgres #default username when you first install PostgreSQL/sign up to PgAdmin4 DB_PASS = somepasswordhere
But really, as you can see in my code on GitHub, you should be using environment variables for those.
The ability to customise almost everything about your bot and use it with your friends is extremely motivating in and of itself. And yes, whilst asynchronous programming can be slightly confusing initially, it's really just all about making sure that nothing is blocking the execution of code, and you will eventually get the hang of it. Whilst I created this bot using Python, a popular alternative is JavaScript, using Node.JS. Go and create your own bot!
