Lexus made sure to arrive ten minutes early to the team meeting on Monday morning. He didn’t really know what to expect, and the last thing he wanted to do was to leave a bad first impression.
As he made his way through the corridors in the upper floors of the computer lab, Lexus couldn’t help but feel out of place. Ever since he arrived at Cambridge as a fresh undergraduate, he’d never ventured beyond the ground floor of the building. He had no reason to, for everything he needed was right there at the ground floor - the common area, the library, and the lecture theatres. Supervisions were always at his college, so he had never needed to enter a professor’s office or lab.
Until now.
Professor Milton’s room was on the second floor, on the East side of the building. For this reason, her room number was “SE26” - “S” for “second” and “E” for “East”. The handy coordinate system embedded into the room number made it easy for Lexus to find.
“Let’s see, it should be here…” Lexus double checked the room number against the number he noted had down on his phone in the prior meeting.
The system let out a big yawn from within his brain. “Yes, you’ve got the right place. Stop being paranoid and just go in.”
Lexus whispered, “System, you’re not helping.”
He took a deep breath and knocked. When he heard a response, he pushed open the door and stepped into the room.
Lexus didn’t know what he was expecting, but he was mildly surprised to see two people sitting next to the large table in the centre of the room, having a chat while sipping on what looked like tea.
I thought I came for a team meeting, not a tea party?
One of the two people was Professor Milton, but the other - a middle aged man - he had never seen before. He did kind of look familiar though… Was he also a computer science professor?
Professor Milton turned her head over at the sound of the door opening. When she saw Lexus, she gave a small smile and raised her hand, beckoning him to also sit down at the table.
“Seeing as your students are starting to file in, I’ll be leaving now,” said the man.
“Thanks Ektor, see you later,” Professor Milton replied, waving him goodbye as he stood up and left through the door from which Lexus just entered, bringing his teacup with him.
Ektor…? Now that he had a name to the face, Lexus finally recalled who the man was.
Ektor. Ektor Wilkinson. Isn’t he a Nobel Prize laureate in Physics? Lexus vaguely remembered seeing him on the news a few years ago. He had made some important discoveries in the field of laser physics, though Lexus didn’t remember anything more specific. He hadn’t bothered to find out anymore at the time, though after today Lexus resolved to learn more about the man and laser physics.
“No need to admire him so much, you’ll be at that level in no time,” the system said.
Lexus didn’t say anything in response as he didn’t want to look weird in front of Professor Milton, but he could not stop the excitement that shone through at the thought of winning something like the Nobel Prize or its computer science equivalent, the Turing Award.
Lexus didn’t really care about the award itself, but he was enchanted with the idea of one day being good enough to do groundbreaking research and change the world with his ideas and inventions.
Is it really achievable? A Nobel Prize laureate must be something like Level 9 or above… Will I ever get there?
Lexus thought about the entity in his brain. If the system says so, it must be possible, right?
After all, he was the chosen one, the person that the system had picked. It was up to Lexus to realise his own potential and become the greatest scientist the world had ever seen.
“Don’t get too complacent though, you’re still nothing more than a slightly competent potato at this stage,” the voice in his head continued.
A potato. A potato?
If left on his own, Lexus might just have retorted back in irritation, but he was interrupted by the words of Professor Milton.
“Morning, Lexus! Would you like a cup of tea?” The professor asked.
Lexus replied, “No, I’m fine, thank you.” He just had a cup of coffee before coming to the lab, and the last thing he wanted was to have to leave to go to the toilet in the middle of the meeting.
Professor Milton nodded. “Make yourself comfortable, the rest of the team should arrive soon.”
As if on cue, the door swung open and in popped a young woman with a smile brighter than the sun. “Morning professor!”
Her eyes landed on Lexus, and her grin only grew wider. “This must be Lexus. I’ve heard so much about you, nice to meet you!”
Lexus was overwhelmed by the sudden enthusiasm that was injected into the room. Luckily, the professor helped him out and said, “Noelle, why don’t you introduce yourself to Lexus?”
Noelle nodded as she took a seat next to Lexus. “Of course! Hi, I’m Noelle Keyes, a third year PhD student working under Professor Milton. My thesis is about improving neural machine translation for low resource languages, which are languages that few people speak.”
“Oh, that sounds cool! I assume that making good use of the data is very important then. How do you plan to do it?” Lexus asked.
If it was even possible, Noelle brightened up even more, clearly excited that someone else was interested in her research. “The specifics would take too long to explain now, but I’m basically going about it from two angles: Augmenting the data itself, as well as tweaking the model architecture. Optimising both means that we can make better use of the small amount of data we have for some of the less popular languages.”
The story has been taken without consent; if you see it on Amazon, report the incident.
For the next five minutes or so, Lexus and Noelle were engrossed in their own conversation while the rest of the team entered the room and sat around the large wooden table. Noelle’s natural enthusiasm turned Lexus’ nervousness into excitement for what the future held for him. This project was going to be so much fun!
“…the project we’re working on is about analysing the data scaling properties of NMT models with respect to changing model architecture, as I’m sure you already know. Oh, the meeting is starting! Anyways, nice to meet you, and here’s to a great time working together,” Noelle said as everyone opened their notebooks and got ready for the team meeting.
Lexus looked around and counted seven people in total. In addition to Professor Milton, Noelle and himself, three other men and one other woman sat around the table in the centre of the room.
The professor kicked off the meeting by introducing Lexus to the rest of the team. “Everyone, this is Lexus. He’s a first year undergraduate, and he’ll be working with Noelle on the NMT data scaling project.” She glanced over at Lexus, signalling for him to say something.
Lexus stood up, put on the best smile he could and formally introduced himself. “Hello everyone! My name is Lexus, and I study computer science. I look forward to meeting all of you and working with the team!” As he sat back down, Lexus was welcomed with a round of applause, with Noelle clapping particularly loudly.
From the corner of his eye, Lexus noticed a bearded man sat diagonally opposite of himself scribbling something down in his notebook. After writing it down, he pushed his glasses back up and sat back to continue listening.
“I don’t think what you said was particularly noteworthy?” the system asked in his head.
For once, Lexus agreed.
The meeting continued as the professor said, “Firstly, I want to congratulate Adi on having his paper accepted by EMNLP as an oral presentation!”
Another round of applause echoed throughout the room.
She turned to Adi, who turned out to be the same bearded man that was scribbling down notes during Lexus’ introduction. The professor asked, “Adi, have you got a powerpoint of your paper ready? It’ll help to have a trial run of your presentation before the actual conference next month.”
Adi nodded and replied, “Yes I do,” standing up to plug his laptop into the projector.
=====
After Adi’s presentation, the rest of the team reported on their progress in the past week, as well as their plans for the coming week. Noelle had just been starting out with the project so she was mostly reading up on literature, though she had made a start with determining the model architectures that she wanted to test. “I think it’ll be a good idea to test a hybrid architecture consisting of a transformer encoder and a LSTM decoder,” she said. “Given its popularity in industry, it would be interesting to compare the sample efficiency of transformers and LSTMs.”
On his notebook, Lexus noted down:
transformer + LSTM hybrid
Lexus thought back to the time where he first came across the concept of transformers. Wanting to understand more about large language models like GPT-3, Lexus went down the rabbit hole of natural language processing and very quickly discovered one of the most important papers in the field, “Attention is All You Need”.
-a few weeks ago-
“Attention is All You Need…" Lexus was hunched over his laptop with his eyes glued to the screen. He scrolled up and down the paper in Zotero, trying to get a sense of what the paper was talking about, but to no avail.
Lexus let out a soft sigh, and rested his head on his hand with his elbow at right angles to the study desk.
“Why can’t I understand anything by skimming through the paper? Look, all I’ve gleamed is that the transformer is this really cool architecture that improves translation performance while decreasing training time… But I have no idea how it actually works!” Lexus said in frustration.
The system replied, “Look, the paper wasn’t meant to be read by a Level 1 computer scientist like you. It’s great that you’re trying to read it, but don’t you think you should put in a little more effort than quickly scrolling through and expecting to miraculously understand everything?”
“Fine, fine,” Lexus grumbled. He scrolled to the top, and started from the first line of the abstract.
Lexus went through the paper line by line, reading each word carefully. Anything he didn’t understand, he would look up online. The system gave no technical guidance to help him learn this difficult topic, but gave encouragement along the way when Lexus felt confused or frustrated. Little by little, Lexus pieced together the story of the transformer.
From what he understood, previous state-of-the-art models used something called recurrent neural networks, which sequentially carry out translation tasks by going through the text one word at a time. For example, for an English-to-German translation task, the model would take in the next English word as well as all of its previously translated German words as input, then spit out the next German word as output. Rinse and repeat until the whole piece of text was translated.
The problem was, computing the next word given a sequence of previous words gets exponentially more expensive as the number of previous words increases. To combat this problem, a common technique was to only look a few words behind, maybe ten words maximum. The obvious problem of this was that the model couldn’t remember things that were mentioned more than a few words ago. For example, if you said “Joe went to school and forgot his pencil case, and so went back home to collect it. He then returned to ___”, the model wouldn’t be able to complete the sentence, because it forgot that Joe was at school!
Enter LSTMs, standing for Long Short -Term Memory. They were a variant of the recurrent neural network, and the main difference between LSTMs and normal recurrent neural networks were that LSTMs had the ability to forget information as the model got further along the piece of text. This meant that if the unimportant information could be forgotten, the model could focus on important information that would still be useful much later down the line. This increased the model’s ability to remember long-term information. The model no longer forgot that Joe was at school.
However, there was another huge problem. Training recurrent neural networks for translation tasks like these were slow. For one, the model required all of the previous translated words as input data, meaning that the text couldn’t be broken up into smaller pieces and be translated on multiple computers or processing cores. In order to train a recurrent neural network, only a single core could be used - which was a waste considering the progress made in parallel computing in recent years. Hell, even a commercial MacBook Air has 8-core GPUs!
The transformer fixed these issues by getting rid of all the recursion. It chose instead to focus on something called attention, which figured out the relative importance of the input words, consisting of the original English words, with respect to the word that was about to be translated. If the model knew which English words or phrases it had to pay attention to when translating the next German word, it wouldn’t need to go through every single word in the English text, as well as every translated word in the German text: it could jump straight to the necessary English phrase and translate it directly.
For the next few hours, Lexus combed through the paper, trying to understand the inner structure of the transformer and noting down all the relevant mathematical formulae. After a single evening, he had reached the last sentence of the paper.
Ding!
---
Computer Science EXP +7
---
“Thank God I have [Increased Processing Speed], otherwise this would have taken me forever to figure out.” Lexus leaned back on his chair and stared blankly onto the computer screen. The words were blurring together and his brain felt like it had just run a marathon, but Lexus didn’t care. In the span of a few hours, Lexus had gone from not knowing anything about transformers at all to understanding a bit about how it worked and, more importantly, why it worked.
In a smug voice the system asked, “Shouldn’t you be thanking me instead?”
-now-
Thanks to past Lexus, which put in the time and effort to read through the paper about transformers, current Lexus sort of understood transformers, and had a vague idea of what LSTMs were about.
He looked down at the words he had just written on his notebook:
transformer + LSTM hybrid
But what was a transformer LSTM hybrid? Lexus had no clue, but he sure wanted to find out!