My 3D Printer Plays Video Games Better Than You

My friends always say I’m trash at Valorant and I play worse than a bot, so I made a robot that plays better than them.

I wanted the robot to literally use a keyboard, mouse, and monitor to play the game so it plays exactly like a human — no memory exploits whatsoever like in most game hacks.

I had no money so I couldn’t buy high-quality servos, power supplies, and other materials — but I did have a 3D printer! If I attach my mouse to the printer nozzle, I can move the mouse by moving the nozzle.

Design 1 - Python Script

My first design was very straightforward.

  1. Take screenshots of Valorant
  2. Look for enemies in the screenshot
  3. Command the 3D printer over USB to aim at the enemy
  4. The 3D printer executes my command
  5. Do this 60 times per second

Easy, right?

1.1 Looking for enemies

Enemies in Valorant have a distinct outline to make them stick out from the environment. You can use simple color filtering for those colors to figure out where they are located on the screen.

screenshot = ImageGrab.grab()
screenshot_array = np.array(screenshot)

lower_red_bound = np.array([0, 0, 140])
upper_red_bound = np.array([100, 100, 255])

# Results in bitmap of pixels that are colored red
red_pixels = cv2.inRange(screenshot_array, lower_red_bound, upper_red_bound)

Assuming there is only one enemy on screen, the average (x,y) coordinate of all red pixels on screen gave me an approximation of the enemie’s “center of mass” and thus where I want to aim.

1.2 Controlling the 3D printer

When you want to print a 3D model, you need to “slice” the model into a “GCode file” which you then execute on the 3D printer. GCode files are like Python turtle programs — step-by-step instructions for the 3D printer.

M104 S185 # Set temperature of nozzle
G0 1 0    # Move nozzle 1u up
G0 0 1    # Move nozzle 1u right
G0 0 -1   # Move nozzle 1u down
G0 -1 0   # Move nozzle 1u left

Typically, you would load the GCode file onto an SD card and insert it into the 3D printer, but you can actually send GCode instructions over USB, one at a time.

import serial

port = "/dev/ttyUSB0"
ser = serial.Serial(port, 9600, timeout=1)

ser.write(b'G0 10 10')

ser.close()

I can use this to control the 3D printer in real time.

1.3 Putting It All Together

Now we can do all of this at 60Hz:

  1. Find the position of the enemy relative to the crosshair.
    1. I take a screenshot of the game.
    2. I use color filtering on the screenshot to find the red outlines of enemies.
    3. Create bounding rectangles for the pixels to find individual enemies.
    4. Find the positions of the enemies relative to the crosshair.
    5. Choose the enemy closest to the crosshair.
  2. Calculate how the 3D printer should move in the.
  3. The 3D printer receives the command and executes it.
    • The 3D printer firmware handles this by itself.

… but this didn’t work at all. The movement was either choppy or delayed no matter how I tuned it. I realized all my problems were caused by the firmware of the 3D printer — the program that’s recieving the GCode commands. The way it handles commands did not suite my project.

Uninterruptable Command Queue

When you send commands to the 3D printer while it is currently in the middle of executing a command, it will queue the new command. Once it finishes the command it’s executing, it will dequeue to get its next instruction. It will not cancel the currently executing command nor is there a way to do that (to my knowledge).

Suppose you send commands that take Δt to finish once every Δt seconds. This should theoretically work fine because as soon as the current command finishes, it will receive a new command with virtually no downtime or latency. But if your command durations or command frequency are desynchronized, then you will have a terrible issue. Either:

  1. The commands take too long to execute, causing the command queue to fill up. Whenever you send a command, it will have to wait until it is reached in the queue. This causes a great lag between when you send the command and when it gets executed.
  2. The commands take too short to execute. When the 3D printer has nothing to execute, it won’t move. These momentary stops cause the 3D printer to have horrible stutters.

Acceleration Curve

The 3D printer will always decelerate to 0 after every command. This means that between every command, it will momentarily stop. Since I was trying to run multiple commands each second, this made it jittery. To solve this, I could run less commands per second but that lower my robot’s reaction speed. I couldn’t find a good balance.

Design 2 - Hand-Written 3D-Printer Firmware

If replace the 3D printer’s firmware with something else, then I can make this robot work better. I decided to write my own.

  1. Take screenshots of Valorant
  2. Look for enemies in the screenshot
  3. Command the 3D printer over USB using GCode to aim at the enemy
  4. Command the 3D printer over USB using my own protocol to aim at the enemy
  5. The 3D printer executes my command
  6. My custom firmware executes my command
  7. Do this 60 times per second

Easy, right?

2.1 Writing New Firmware for 3D Printers

My 3D printer is a Monoprice Maker Select V2 which has a Melzi 2 control board.

Melzi 2 board

Despite how it looks, this board is literally just an Arduino. It has an ATMEGA1284P processor which is a Sanguino microcontroller. Sanguinos are high-performance chips that are compatible with Arduino software.

Initially, there’s no way to overwrite the default 3D printer firmware. I had to follow this guide to install a bootloader, which lets me upload C++ code to the Melzi using the Arduino IDE just like any other Arduino.

For this blog post, I’m using pseudocode to make thing simpler. If you want to learn about writing firmware for 3D printers in detail, read this post.

The code comes in two parts — getting the data from USB, and then moving the motors according to the given state.

int xDisplacement, yDisplacement;

while true {
  xDisplacement, yDisplacement = getDataFromUSB();
  moveMotors();
}

2.2 New Communication Format

Since we’re getting rid of the 3D printer firmware, we can drop g-code entirely. I switched to a new format that’s a lot easier and faster to use.

Simple: 3 bytes. The first two are 1-byte signed integers which denotes the displacement the enemy is from the crosshair. The third bit is 0, and is used for alignment. The first two bytes can’t be 0 so we can’t get them confused.

bool correctingError = false;
void updateError() {
  if (correctingError && Serial.available()) {
    byte data = Serial.read();
    if (data == END_BYTE) {
      correctingError = false;
    }
  }
  if (Serial.available() >= 3) {
    byte data[3];
    Serial.readBytes(data, 3);
    if (data[2] == END_BYTE) {
      updateVars(data[0] - 128, data[1] - 128);
    } else {
      correctingError = true;
    }
  }
}

2.3 Moving Servos

We can control each servo using a set of digital pins:

  1. Enable - Lock/unlock the motor
  2. Direction - Chooses the direction the motor will move (0=clockwise, 1=counterclockwise)
  3. Step - Toggling this moves the motor 1 “step”
  4. Step mode - An n-bit signal that decides how big a step is

For example, here’s some pseudocode that will move the motor as fast as possible:

CW, CCW = 0, 1
digitalOutput(DIRECTION_PIN, CCW)
stepState = False
for _ in [0..numSteps]:
  digitalOutput(STEP_PIN, stepState)
  stepState = ~stepState # Invert state

The faster you can toggle the state of STEP_PIN, the faster the stepper motor goes. Additionally, this means our program must be in the form of a very efficient loop. We must toggle the state regularly or else the stepper motor will jitter.

Irrelevant rant There’s one thing I want to note about development on a Melzi board that frustrates me dearly. Stepper motors move in units called “steps” which is dependent on the specific design of the given motor, and is typically 30 degrees. In order to make the motor move a step, you need to alternate the STEP_PIN. If I want to make the motor move 5 steps, I alternate the STEP_PIN 5 times. Stepper motors can also move in half, quarter, eigth and potentially up to 256th of a step depending on the motor. Larger steps make it move faster, and smaller steps allow more precision.

Stepper driver. It has pins labelled “mode 0”, “mode 1”, “mode 2”, “mode 3”

Whether the stepper driver will make the motor move in full, half, quater, etc steps depends on what mode it is in. That can be decided by using the mode pins. The melzi pinout doesn’t have any pins to choose the mode because the drivers are hard wired to stay on the 1/16th-step mode. Melzi stepper driver diagram

I really wanted to use full steps for large, sweeping motions and then switch to smaller steps for small aim adjustments but because they hard wired it, I just had to make do. The speed of the motor depends on how fast you can alternate its STEP_PIN, so I had to make sure performance was a priority in my code. Moving the stepper motor is typically a blocking statement so so the microcontroller has enough processing power to alternate the STEP_PIN in a stable way. I found all the libraries for Stepper motors extremely limiting because they’re suited for constant acceleration.

2.4 Proportional Integral Derivative Control

We can state this problem in terms of error correction. The number of pixels the robot is away from the enemy is the error, and our goal is to make this error zero. When the error is zero, it means our crosshair is right on top of the enemy. PID is a very classic algorithm used in robotics and I used it all the time in my high school’s FIRST robotics club, but I didn’t feel this problem needed a full-blown PID algorithm. The output of PID tells me what speed I need to set the motor to quickly and smoothly move the motor to a position, but I didn’t have such direct control over the speed. As shown in the previous section, all I could do is move it one step at a time.

I had to take an alternative approach. “Inverse P” is not a real control system, but it’s the most natural solution to this problem under the given restrictions.

step = 0
while True:
    error = get_error()
    period = c * 1. / error
    if step % period == 0:
      moveMotorOneStep()
    step += 1

The higher the error, the higher the frequency the motor would be moved one step, and thus the faster the motor will move.

2.5 Putting it all together

  1. Python code running on computer
  2. Take screenshot of game and find location of enemy exactly the same as in part 1
  3. Send the displacement between the crosshair and enemy over USB to the 3D printer
  4. New firmware running on 3D printer
  5. Recieve the data from USB
  6. Calculate the new speed at which it should move the motors
  7. Move the motors

This worked! But not that well…

Final Design — Enemy Detection with an FPGA

In my freshman year of college, I did in an internship at an aerospace company where I worked on developing a circuit board using an FPGA.

After profiling my system, I realized that the slowest part was the python code… unsurprisingly. Instead of having to take a screenshot, I wanted to create a circuit that reads the video signal from the computer and finds enemies faster than they can actually show up on your monitor.

gantt
    dateFormat SS
    axisFormat %M
    Task A : 10m
    Task B : 5m

I still had the Arduino Vidor from that internship, so that’s what I used for this project.

What’s an FPGA?

An FPGA is a special circuit that can can be transformed into other circuits. It’s literally like magic. You can write code in a “hardware description language” to describe a physical circuit, upload the code to the FPGA, and it will pretend to be that physical circuit. It’s typically used for developing circuits so you don’t have to fabricate a new circuit board or chip every time you want to test something.

Devices like theArduino Vidor are cool because they has an FPGA and a normal CPU on one board. Just like how your CPU and GPU work together, you can make the CPU and FPGA work together. You can offload certain tasks to the FPGA to make it faster.

How Does VGA Work?

VGA is a type of video cable for connecting your computer to your monitor. It serves the same function as HDMI or DisplayPort. I wanted to connect the VGA signal from my computer to my circuit because I felt it was the easiest to work with. HDMI and DisplayPort signals are hard to decode and often require a dedicated chip for that. VGA is a very primitive video format and is effectly transmitted as “plain text”.

The VGA signal essentially sends you the color of each pixel one at a time. It has 3 pins sending the RGB value of the pixel, and the other pins are a clock that tell when it wants to send the next pixel.

The Circuit

I’m too lazy to finish writing everything below this line

Instead of using a script running on my computer which could potentially be detected, I wanted to create an external device to analyze the video output of my computer. I started by converting an analog VGA signal to a digital signal and reading it with an Arduino Vidor. The VGA pins are connected to the FPGA which takes advantage of VGA’s straightforward encoding format to calculate the “center point” of all red pixels in the given frame. It then saves this in the Vidor’s shared memory between the FPGA and ARM chip. I then use the ARM chip to send instructions to the 3D printer over UART. I know it’s possible to connect the FPGA directly to the UART chip but I’m having trouble with it for some reason so I had to do it in a needlessly complex way. I’m currently working on simplifying it using an HDMI decoder to do all the hard work for me. I orignally used VGA because I heard that unencypting HDMI was a nightmare and would be impossible for me to do on an FPGA (while VGA is a very simple format to parse), but I had not considered the fact that there exists a perfectly good commercial solution that does that for me The current plan is to use a breakout board to get the 40-pin output plugged into my FPGA and then interpret the data there. I can’t find any resources on the specifics of which pins are which, but it should be a pixel clock similar to VGA. There is also the possibility that the 40-pin FPC breakout board is not bidirectional. I’ve only seen videos of people using it to output through the 40-pin connector and not breakout a 40-pin connector to read using an arduino.

I expect that the video format is unsophisticated and I can easily parse it using the FPGA.

IC Design

Beyond

TODO