Harbola DataScience

Answer, Read and Post Coding related Questions!

How to Create a Virtual Pen Using Python !

Imagine how interesting it would be if you don’t have to touch anything. Just wave your pen or any object on air and your web camera will detect it and write it on your screen. You need only a camera. That’s it. And will will code everything from scratch. And what your reaction would be if I say you don’t even need machine learning or deep learning to acheive this?

It can change the way of teaching and revolutionize writing & drawing. It need some more advancement and improvement and can be taken to next level.

In this article I am gonna walk you with the implementation of this virtual pen using python and will also make it easy for you to do it yourself.

Demonstration of virtual pen using Python

It is detecting yellow color and drawing lines wherever I move this pen or anything that is of yellow color.

Drawback in Virtual pen

As it is detecting the color, therefore there is a little room for error because if you wear or have any object that is of same color that your software is detecting, it may misinterpret and draw according to that. Which is something that we don’t want.


This thing can be handled if you change the color that you want your software to detect hyperparameter tuning. We will also see that in this article.

Take a pen whose color don’t match with color of any object around you.

Implementation of virtual pen using python opencv

This involves two step.

  • detecting your object
  • drawing lines as it moves

Importing required libraries and modules

install cv2 using pip install opencv-python

Loading file using numpy method

You need penval.npy file to remove every object except the object that you want you system to detect. You need to do some hyperparameter tuning. For everyone the value inside this penval file changes.

Here I am just showing you detecting blue color objects. You have to adjust the trackbars below to change the color you want to detect. This setting will be for blue color object. If your pen is of blue color you can use this. Else change it so that on black screen only your pen is visible. Here my headphone are visible. So if I use this setting for my code…lines will be drawn as I move my headphones.

Hope you got the intiution behind it.

Save this file into a .npy file which will use further to draw line. I will provide you code to get this dialogue box to adjust the setting.

Now your object is detected.

Drawing lines as object moves

Now our main code starts. We will open our web cam with VideoCapture(), name the window, initialize values and assign some threshold values.

You can face some errors. Take help of this article to solve them.

Create a infinite while loop until web cam is opened. Read the frame, convert BGR image to HSV(Hue Saturation Value), load the values that you assigned to .npy file on detection part. If you don’t have .npy file you can assign your own custom values numpy array method.


Morphology is a broad set of image processing operations that process images based on shapes. In a morphological operation, each pixel in the image is adjusted based on the value of other pixels in its neighborhood. Dilate, erode, reconstruct, and perform are some morphological operations.

Finding the shape of the outer surface

If you don’t know about contour or what they does I recommend you going through this project. In this we have talked about them.

Creating a virtual eraser

we will display “Clearing Canvas” and clear the screen. You can also print any other text on screen while erasing like ” please wait…” or “erasing”

Final touch

now every difficult and important tasks are completed. Now just we need to show the web cam screen. You can change the name of web cam window from “image” to ” virtual pen system”.

Chitranshu Harbola

Self taught programmer, Web Developer and an aspiring Machine learning engineer cum Data Science student

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Back to top