Sports Analytics: Tutorial To Build Ball Detection AI Model

Sumit Singh -

Ever wondered how cricket ball is detected and how their trajectories are presented to us in the form of lines, ever wondered even these fastballs are detected easily, and how they are detected, In this tutorial we will teach you how can you make these tracking programs that will do the work for you.

Cricket, often celebrated for its unpredictable twists and turns, has grappled with contentious umpiring decisions, stirring debates and controversies. The fallibility of the human eye, despite rigorous training, has prompted the integration of cricket ball detection technologies.

This technological leap aims to redefine the dynamics of the sport by mitigating human errors and introducing a more objective approach to decision-making. In this blog, we embark on an exploration of ball tracking systems, delving into their potential to augment accuracy, minimize controversies, and shape the future landscape of cricket.

Table of Contents

  1. The Current Landscape
  2. The Imperative for Change
  3. How Ball Detection Operates
  4. Use Cases of Ball Tracking Systems in Sports
  5. Different Approaches to Ball Tracking Systems
  6. Implementation - Developing Your First Ball Tracking System
  7. Conclusion

The Current Landscape

Cricket, like any other sport, heavily relies on the judgments of on-field umpires. While these officials boast considerable experience and expertise, the frenetic pace of the game, particularly in T20 formats, occasionally leads to unavoidable errors. Umpires are tasked with making split-second decisions, often in high-pressure situations, leaving room for controversy and discourse.

The Imperative for Change

To address the challenges tied to human judgment, the cricketing community is increasingly embracing technology. Ball tracking systems, leveraging advanced computer vision and artificial intelligence, stand poised to revolutionize the sport.

By automating the ball detection process, these systems offer a more objective and precise means of decision-making, reducing the influence of human error and elevating the overall fairness of the game.

How Ball Detection Operates

A ball tracking system deploys sophisticated cameras strategically positioned around the cricket field to capture every nuance of the ball's movement. These cameras transmit data to a computer system equipped with robust algorithms capable of dissecting the trajectory, speed, and direction of the ball. The result is a real-time visualization of the ball's path, equipping umpires with an additional layer of information to make more informed decisions.

Advantages of Ball Tracking:

  1. Precision: The foremost benefit of ball detection systems lies in their capacity to furnish precise and dependable data. This precision significantly diminishes the occurrence of contentious decisions, fostering a more equitable and transparent game.
  2. Equity: By minimizing human error, ball tracking systems uphold the principles of fair play, ensuring that match outcomes are determined by players' skill and performance rather than umpiring decisions.
  3. Enhanced Viewer Experience: The introduction of ball tracking technology enriches the viewer experience, offering fans a deeper comprehension of the game. Detailed graphics and insights into the ball's trajectory contribute to a more engaging and informative spectacle.
  4. Umpire Aid: Ball tracking systems are designed to complement on-field umpires, not replace them. Umpires can utilize the technology as a tool to enhance decision-making accuracy, ultimately enriching the officiating process.

Let's Implement our own Ball detection classifier

Use Cases of Ball Tracking Systems in Sports

Ball tracking systems play a pivotal role in sports analytics. Here are a few use cases in cricket:

  1. Critical Decision Making: Assists in decisions like Leg Before Wicket (LBW), determining whether the ball has pitched inside or outside the stumps.
  2. Identifying Strong and Weak Zones: Analyzing videos to generate heat maps helps teams identify a batsman's strong and weak zones, aiding strategic planning.

Different Approaches to Ball Tracking Systems

Tracking a fast-moving cricket ball presents challenges due to its high speed. Two simple approaches are discussed:

  1. Sliding Window: Divides the image into smaller patches, classifying each patch to detect the ball. It's effective but computationally expensive.
  2. Segmentation by Color: Utilizes the known color of the ball to reduce the number of patches for classification based on color similarity.

Implementation - Developing Your First Ball Tracking System

Let's start coding! The tutorial covers the following steps:

  1. Reading video frames and saving them as images.
  2. Scene detection to identify frames containing the pitch.
  3. Implementing a color-based segmentation approach for a single frame.
  4. Extracting patches, building an image classifier, and evaluating it.
  5. Applying the developed system to each frame in the video for ball tracking.

Dataset
You can access the dataset from here

step1: Read the video files

import cv2
import numpy as np
import imutils

video='28.mp4'

# Create a VideoCapture object and read from input file
# If the input is the camera, pass 0 instead of the video file name
cap = cv2.VideoCapture(video)
cnt=0

# Check if camera opened successfully
if (cap.isOpened()== False): 
  print("Error opening video stream or file")

ret,first_frame = cap.read()

# Read until video is completed
while(cap.isOpened()):
    
  # Capture frame-by-frame
  ret, frame = cap.read()
     
  if ret == True:
    
    #removing scorecard
    roi = frame[:800,:]
    
    #cropping center of an image
    thresh=600
    end = roi.shape[1] - thresh
    roi = roi[:,thresh:end]
    
    cv2.imshow("image",roi)
    # Press Q on keyboard to  exit
    if cv2.waitKey(25) & 0xFF == ord('q'):
      break

    cv2.imwrite('frames/'+str(cnt)+'.png',roi)
    cnt=cnt+1

  # Break the loop
  else: 
    break

cv2.destroyAllWindows()

Read the frames:

import matplotlib.pyplot as plt
import cv2
import numpy as np
import os
import re

#listing down all the file names
frames = os.listdir('frames/')
frames.sort(key=lambda f: int(re.sub('\D', '', f)))

#reading frames
images=[]
for i in frames:
    img = cv2.imread('frames/'+i)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    img = cv2.GaussianBlur(img,(25,25),0)
    images.append(img)

images=np.array(images)

Extracting the frames

nonzero=[]
for i in range((len(images)-1)):
    
    mask = cv2.absdiff(images[i],images[i+1])
    _ , mask = cv2.threshold(mask, 50, 255, cv2.THRESH_BINARY)
    num = np.count_nonzero((mask.ravel()))
    nonzero.append(num)
    
    
x = np.arange(0,len(images)-1)
y = nonzero

plt.figure(figsize=(20,4))
plt.scatter(x,y)

The outlier in the plot indicates the frame number during which the scene changes. So, fix the threshold for obtaining the frames before a scene change:

threshold = 15 * 10e3
for i in range(len(images)-1):
    if(nonzero[i]>threshold): 
        scene_change_idx = i
        break
        
frames = frames[:(scene_change_idx+1)]


Now, we have obtained the frames that contain a pitch. Next, we will implement a segmentation approach that we discussed earlier in the article. Let’s carry out all the steps of the approach for only a single frame now.

We will read the frame and apply Gaussian blur to remove noises in an image:

img= cv2.imread('frames/' + frames[10])
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
gray = cv2.GaussianBlur(gray,(25,25),0)

plt.figure(figsize=(5,10))
plt.imshow(gray,cmap='gray')

As the color of the ball is known, we can easily segment the white-colored objects in an image. Here, 200 acts as a threshold. Any pixel value below this 200 will be marked as 0 and above 200 will be marked as 255.

_ , mask = cv2.threshold(gray, 200, 255, cv2.THRESH_BINARY)

plt.figure(figsize=(5,5))
plt.imshow(mask,cmap='gray')

As you can see here, the white-coloured objects are segmented. The white color indicates white colored objects and black indicates the rest of the colors. And that’s it! We have separated the white-coloured objects from the others.

Now, we will find the contours of segmented objects in an image and draw the contours of original image

image, contours, _ = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
img_copy = np.copy(gray)
cv2.drawContours(img_copy, contours, -1, (0,255,0), 3)
plt.imshow(img_copy, cmap='gray')

Next, extract the patches from an image using the contours:

!rm -r patch/*

num=20
cnt=0
for i in range(len(contours)):
    x,y,w,h = cv2.boundingRect(contours[i])
    
    numer=min([w,h])
    denom=max([w,h])
    ratio=numer/denom

    if(x>=num and y>=num):
        xmin, ymin= x-num, y-num
        xmax, ymax= x+w+num, y+h+num
    else:
        xmin, ymin=x, y
        xmax, ymax=x+w, y+h

    if(ratio>=0.5 and ((w<=10) and (h<=10)) ):    
        print(cnt,x,y,w,h,ratio)
        cv2.imwrite("patch/"+str(cnt)+".png",img[ymin:ymax,xmin:xmax])
        cnt=cnt+1


It’s time to build an image classifier to identify the patch containing the ball.

Reading and preparing the dataset:

import os
import cv2
import numpy as np
import pandas as pd
folders=os.listdir('data/')

images=[]
labels= []
for folder in folders:
    files=os.listdir('data/'+folder)
    for file in files:
        img=cv2.imread('data/'+folder+'/'+file,0)
        img=cv2.resize(img,(25,25))
        
        images.append(img)
        labels.append(int(folder))

images = np.array(images)
features = images.reshape(len(images),-1)

Split the dataset into train and validation:

from sklearn.model_selection import train_test_split
x_tr,x_val,y_tr,y_val = train_test_split(features,labels, test_size=0.2,
stratify=labels,random_state=0

Build a baseline model for identifying the patch containing ball

from sklearn.ensemble import RandomForestClassifier 
rfc = RandomForestClassifier(max_depth=3) 
rfc.fit(x_tr,y_tr)

Evaluate the model on the validation data:

from sklearn.metrics import classification_report
y_pred = rfc.predict(x_val)
print(classification_report(y_val,y_pred))

Repeat the similar steps for each frame in a video followed by classification:

!rm -r ball/*
ball_df = pd.DataFrame(columns=['frame','x','y','w','h'])

for idx in range(len(frames)):
    
    img= cv2.imread('frames/' + frames[idx])
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    gray = cv2.GaussianBlur(gray,(25, 25),0)
    _ , mask = cv2.threshold(gray, 200, 255, cv2.THRESH_BINARY)
    image, contours, _ = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    
    !rm -r patch/*

    num=20
    cnt=0
    df = pd.DataFrame(columns=['frame','x','y','w','h'])
    for i in range(len(contours)):
        x,y,w,h = cv2.boundingRect(contours[i])

        numer=min([w,h])
        denom=max([w,h])
        ratio=numer/denom

        if(x>=num and y>=num):
            xmin, ymin= x-num, y-num
            xmax, ymax= x+w+num, y+h+num
        else:
            xmin, ymin= x,y
            xmax, ymax= x+w, y+h

        if(ratio>=0.5):    
            #print(cnt,x,y,w,h,ratio)
            df.loc[cnt,'frame'] = frames[idx]
            df.loc[cnt,'x']=x
            df.loc[cnt,'y']=y
            df.loc[cnt,'w']=w
            df.loc[cnt,'h']=h
            
            cv2.imwrite("patch/"+str(cnt)+".png",img[ymin:ymax,xmin:xmax])
            cnt=cnt+1
    
    
    files=os.listdir('patch/')    
    if(len(files)>0):
    
        files.sort(key=lambda f: int(re.sub('\D', '', f)))

        test=[]
        for file in files:
            img=cv2.imread('patch/'+file,0)
            img=cv2.resize(img,(25,25))
            test.append(img)

        test = np.array(test)

        test = test.reshape(len(test),-1)
        y_pred = rfc.predict(test)
        prob=rfc.predict_proba(test)

        if 0 in y_pred:
            ind = np.where(y_pred==0)[0]
            proba = prob[:,0]
            confidence = proba[ind]
            confidence = [i for i in confidence if i>0.7]
            if(len(confidence)>0):

                maximum = max(confidence)
                ball_file=files[list(proba).index(maximum)]

                img= cv2.imread('patch/'+ball_file)
                cv2.imwrite('ball/'+str(frames[idx]),img)

                no = int(ball_file.split(".")[0])
                ball_df.loc[idx]= df.loc[no]
            else:
                ball_df.loc[idx,'frame']=frames[idx]

        else:
            ball_df.loc[idx,'frame']=frames[idx]

Have a glance at the frames containing the ball along with the location:

ball_df.dropna(inplace=True)
print(ball_df)

Next, we will draw the bounding box around the frames that contain the ball and save it back to the folder:

files = ball_df['frame'].values

num=10
for idx in range(len(files)):
    
    #draw contours 
    img = cv2.imread('frames/'+files[idx])
    
    x=ball_df.loc[idx,'x']
    y=ball_df.loc[idx,'y']
    w=ball_df.loc[idx,'w']
    h=ball_df.loc[idx,'h']
    
    xmin=x-num
    ymin=y-num
    xmax=x+w+num
    ymax=y+h+num

    cv2.rectangle(img, (xmin, ymin), (xmax, ymax), (255,0,0), 2)
    cv2.imwrite("frames/"+files[idx],img)

Let’s convert back the frames into a video now:

frames = os.listdir('frames/')
frames.sort(key=lambda f: int(re.sub('\D', '', f)))

frame_array=[]

for i in range(len(frames)):
    #reading each files
    img = cv2.imread('frames/'+frames[i])
    height, width, layers = img.shape
    size = (width,height)
    #inserting the frames into an image array
    frame_array.append(img)

out = cv2.VideoWriter('28.mp4',cv2.VideoWriter_fourcc(*'DIVX'), 25, size)
 
for i in range(len(frame_array)):
    # writing to a image array
    out.write(frame_array[i])
out.release()

Result:

Conclusion:

  1. Technological Leap in Cricket: The integration of ball tracking systems, driven by computer vision and artificial intelligence, marks a transformative shift in cricket to mitigate human errors and introduce objectivity in decision-making.
  2. Addressing Umpiring Challenges: Human errors in high-pressure situations prompt the need for change. Ball tracking systems automate detection, offering a more precise and objective approach, reducing controversies.
  3. Operational Dynamics: These systems use strategically positioned cameras to capture ball movement, providing real-time data on trajectory, speed, and direction, aiding umpires in making more informed decisions.
  4. Implementation Tutorial: The blog provides a practical guide on building a ball detection classifier using Python, TensorFlow, and Keras, encouraging hands-on exploration.
  5. Use Cases in Sports Analytics: From critical decision-making (LBW) to strategic planning based on batsmen's strong and weak zones, ball tracking systems play a pivotal role in sports analytics.
  6. Future Impact: As cricket evolves, the integration of advanced systems promises to shape a future where accuracy, transparency, and engagement define the sport for both players and fans.