PenaltyVisual – Penalty Direction Prediction from Kicker Motion
Computer vision on penalty-kick footage: YOLO11 tracks ball and kicker, and a simple physics model pins down the kick frame to within one frame (40 ms), even when the leg hides the ball.
- Role
- Sole developer (dataset, models, evaluation)
- Organization
- Personal project
- Period
- 2025 – 2026
- computer vision
- sports analytics
- YOLO11
- object detection
- video
- physics-based modelling
- PyTorch
Context
A personal project with a long-term goal: from the video of a penalty, predict where the ball will go before it gets there. To do that, a model has to look at the right moment: the run-up and the instant the foot meets the ball. Everything downstream depends on finding that instant.
I built a dataset of 383 penalty clips from public match footage and annotated each one by hand: shot side and height, kicking foot, run-up speed, feints, camera angle. Then I built the pipeline around it.
The problem
At 25 frames per second, one frame is 40 ms, and a struck ball travels about a metre in that time. Being “close” to the kick frame is not good enough: two frames late, the clip already shows the ball in flight.
Broadcast footage makes this hard:
- The ball is tiny: 15–20 pixels across in a 720p frame.
- The camera pans and zooms throughout the run-up, so a ball at rest still moves on screen.
- The contact itself is often invisible. In many clips the kicking leg hides the ball for two to seven frames, exactly when the kick happens.
Approach
Detection. I fine-tuned YOLO11 (nano and medium) from COCO weights on two classes, ball and kicker. It runs on every frame. For the ball, each frame keeps the detection closest to the previous position.
Kick frame: a physical model instead of a heuristic. My first version took the frame where the ball and the kicker are closest. It is simple, and often right, but it can be wrong by more than half a second: the closest point between two bounding-box centres depends on the kicker’s posture, not on when the ball moves.
So I went back to the physics of the event:
- Remove the camera. Phase correlation between consecutive frames measures the pan; subtracting it gives the ball’s motion on the pitch, not on the screen. A ball at rest now really sits still.
- Before the kick, the ball is at rest on the spot. After it, it flies in a near-straight line at near-constant speed.
- Fit the flight and extend it backwards. A line through the first detections after the ball reappears, extended back to the spot, gives the moment it left the spot, even if the contact itself was hidden by the leg.
In other words, the kick is the single impulse that takes the ball from zero velocity to flight, and you can find it from what happens after it.
When the physical model cannot be applied (the ball is never seen at rest, or too few detections after the kick), the pipeline falls back to the closest ball–kicker frame.
Prediction (prototype). From the kick frame, the pipeline cuts two windows: 32 frames of run-up and 16 frames around the kick. It removes the background (temporal median of the clip, with only the kicker’s box pasted back) and encodes each window with MViT-v2, a video transformer pre-trained on Kinetics-400. A small two-branch classifier, fed with these embeddings, predicts left, centre or right.
My role
Everything: the dataset (download, cutting, a small web tool to annotate 383 clips), YOLO fine-tuning, the timing method and its evaluation, and the prediction prototype.
Results
Detection. On the validation split, the five YOLO11 runs reach a mAP@50 between 0.81 and 0.89 (ball and kicker together). The nano model used for the timing analysis scores 0.88.
Kick frame. I checked the kick frame by eye on 12 clips that I had not used to design the method:
| Method | Exact frame | Within ±1 frame | Worst error |
|---|---|---|---|
| Closest ball–kicker (first version) | 5 / 12 | 8 / 12 | 15 frames (0.6 s) |
| Peak of ball acceleration | 4 / 12 | 8 / 12 | 11 frames |
| Flight extended back to the spot | 6 / 12 | 12 / 12 | 1 frame (40 ms) |
The physical model can be applied to 23 of 40 clips in a sample; the others use the fallback. The reference frames come from my own visual check, at broadcast resolution, so they carry about ±1 frame of uncertainty themselves.
On real footage I can show. The training clips are TV broadcasts, which I can’t republish, so the frames below come from a free stock video: a close-up penalty filmed from behind the kicker.

Footage: Pexels, Pexels License. Frames cropped and annotated by me.
At this distance the physics needs no extrapolation: the ball sits still on the spot until frame 36, and at frame 37 it is found displaced. That is the kick, the same frame I see by eye. Two honest caveats. The boxes come from the generic YOLO11m (COCO “sports ball”), not from my fine-tuned models: trained on TV footage, where the ball is a few pixels wide, they barely find a ball this large. And on a second free video, filmed from the stands, the fine-tuned model tracks ball and kicker up to the contact but never detects the ball in flight, a small blurred streak. Both point to the same next step: training data with more camera distances, and blurred balls in flight.
Prediction. Not there yet. The only trained version of the classifier reaches 44% accuracy on a validation set of 27 clips. Always guessing the most common side would score 45%. The model never predicts “left” correctly. With a few hundred clips, generic action embeddings don’t seem to capture where the kicker is aiming. That’s why I keep this part labelled as a prototype.
Stack
Python, PyTorch, Ultralytics YOLO11, OpenCV (phase correlation, optical flow), torchvision MViT-v2, scikit-learn, Weights & Biases, Roboflow.
What I’d do next
- Bring the physical model fully into the pipeline and extend the manual check from 12 clips to a proper annotated test set.
- Replace generic video embeddings with features that carry intent: body pose at the moment of contact (hip and plant-foot orientation), with the kick frame as the anchor.
- Remove the hand-annotated inputs from the classifier: some, like whether the shot was on target, would not be available before the kick.
- Record my own footage, so the project can show real frames, not only charts.