UPDATE: Version 2 of my Lane Detection System is up! Check out my Curved Lane Letector here!Introduction
In any driving scenario, lane lines are an essential component of indicating traffic flow and where a vehicle should drive. It's also a good starting point when developing a self-driving car! In this project, I'll be showing you how to build your own lane detection system in OpenCV using Python. Here's the structure of our lane detection pipeline:
- Reading Images
- Color Filtering in HLS
- Region of Interest
- Canny Edge Detection
- Hough Line Detection
- Line Filtering & Averaging
- Overlay detected lane
- Applying to Video
OpenCV is a very popular and well-documented library for computer vision. It's a great tool for any machine vision or computer vision application. If you haven't already installed it, check out the documentation and follow the installation tutorials.Jupyter Notebooks
Ipython/Jupyter Notebooks are a fantastic tool for experimenting and testing code. You can quickly modify programs and run specific functions, making it really easy to develop a prototype. You can install Jupyter from your terminal using the following command:
1. Reading Images
pip install jupyter
When developing an image processing pipeline, you need to read some example inputs that you can test your pipeline on. You can read single images with the following method in OpenCV:
img = cv2.imread("imageDirectory.jpg")
Keep in mind it reads images in the BGR colorspace, and not the RGB colorspace. If you ever need to convert from one colorspace to another (BGR to RGB in this example), you can use the following method in OpenCV:
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # converts from BGR to RGB
However, we'll be using matplotlib in our program to read and display images, since it works very nicely with Jupyter environments. You can install matplotlib using:
pip install matplotlib
We'll be reading 6 different test images in a folder named
test_images/. Here's our code to read the test images in that directory:
imageDir = 'test_images/' imageFiles = os.listdir(imageDir) imageList =  #this list will contain all the test images for i in range(0, len(imageFiles)): imageList.append(mpimg.imread(imageDir + imageFiles[i]))
and the following function to display our test images:
def display_images(images, cmap=None): plt.figure(figsize=(40,40)) for i, image in enumerate(images): plt.subplot(3,2,i+1) plt.imshow(image, cmap) plt.autoscale(tight=True) plt.show()
when we call the function
display_images with the input
imageList, you should see this in your output cell:
Now let's move on to the actual pipeline.2. Color Filtering
Lane lines usually aren't colored green or blue or brown. They're generally colored white or yellow. We can filter out any unnecessary colors so that we can make sure our pipeline is processing only lane lines.
Color filtering in RGB or BGR colorspaces is unnecessarily difficult, however. We can use different colorspaces, or representations of color, to easily filter out extraneous colors. One such colorspace that makes color filtering really easy is the HLS colorspace, which stands for Hue, Lightness, and Saturation. Each pixel dimension's value is between 0-255.
We can convert images in the
BGR colorspace to
HLS like this:
hls = cv2.cvtColor(image, cv2.COLOR_RGB2HLS)
To get the yellow lane lines, we'll be getting rid of any pixels with a Hue value outside of 10 and 50 and a high Saturation value. To get the white lane lines, we'll be getting rid of any pixels that have a Lightness value that's less than 190.
We can then add the filtered yellow and white lane lines into a single image.
Here's the function to filter out the lane lines and output them:
def color_filter(image): #convert to HLS to mask based on HLS hls = cv2.cvtColor(image, cv2.COLOR_RGB2HLS) lower = np.array([0,190,0]) upper = np.array([255,255,255]) yellower = np.array([10,0,90]) yelupper = np.array([50,255,255]) yellowmask = cv2.inRange(hls, yellower, yelupper) whitemask = cv2.inRange(hls, lower, upper) mask = cv2.bitwise_or(yellowmask, whitemask) masked = cv2.bitwise_and(image, image, mask = mask) return masked
to apply this to a list of images, we can use the
filtered_img = list(map(color_filter, imageList))
When we display the images, you should see the following in your output cell:
As you can see in the previous images, there are still some portions of the image that met the threshold, but weren't lane lines. We can narrow down where we're looking for lane lines by masking out a region of interest using the following function:
def roi(img): x = int(img.shape) y = int(img.shape) shape = np.array([[int(0), int(y)], [int(x), int(y)], [int(0.55*x), int(0.6*y)], [int(0.45*x), int(0.6*y)]]) #define a numpy array with the dimensions of img, but comprised of zeros mask = np.zeros_like(img) #Uses 3 channels or 1 channel for color depending on input image if len(img.shape) > 2: channel_count = img.shape ignore_mask_color = (255,) * channel_count else: ignore_mask_color = 255 #creates a polygon with the mask color cv2.fillPoly(mask, np.int32([shape]), ignore_mask_color) #returns the image only where the mask pixels are not zero masked_image = cv2.bitwise_and(img, mask) return masked_image
We'll apply this to our test images using:
roi_img = list(map(roi, filtered_img))
display_images(roi_img) should result in the output cell displaying this:
Now that we have images in which the lane lines have been isolated, we can compute the edges of the lane lines. This can easily be done using CannyEdgeDetection. The idea behind Canny Edge Detection is that pixels near edges generally have a high gradient, or rate of change in value. We can use this to our advantage and use it to detect lane lines. Note: make sure you convert the colorspace to grayscale before applying Canny Edge Detection.
You can apply a Canny Edge Detector using the following method:
cv2.Canny(gray, 50, 120)
where 50 and 120 are thresholds for the hysteresis procedure. Here's the code I used to convert to grayscale and extract the edges:
def grayscale(img): return cv2.cvtColor(img, cv2.COLOR_RGB2GRAY) def canny(img): return cv2.Canny(grayscale(img), 50, 120) canny_img = list(map(canny, roi_img)) display_images(canny_img,cmap='gray')
You should see the following in your output cell:
This is the hardest part of the project. We need to determine the lane lines from the canny edges we previously detected.
How can you detect lines? We'll be computing the lines using
cv2.HoughLinesP(), and then filtering out the lines. We'll start by removing lines that are outside a determined slope range, and location. Additionally, we will filter the lines into their own corresponding lanes by seeing whether or not the filtered lines have positive or negative slope.
Now we have the lines that correspond to their lane; how do we combine them into a single line? One way to approach it is by using the slope and intercept of the lines. We'll be calculating the slope and intercept of each of the lines in a particular lane, then averaging the slope and intercepts to produce one line. We'll do this to both lanes.
Here's the code to do this:
rightSlope, leftSlope, rightIntercept, leftIntercept = ,,, def draw_lines(img, lines, thickness=5): global rightSlope, leftSlope, rightIntercept, leftIntercept rightColor=[0,255,0] leftColor=[255,0,0] #this is used to filter out the outlying lines that can affect the average #We then use the slope we determined to find the y-intercept of the filtered lines by solving for b in y=mx+b for line in lines: for x1,y1,x2,y2 in line: slope = (y1-y2)/(x1-x2) if slope > 0.3: if x1 > 500 : yintercept = y2 - (slope*x2) rightSlope.append(slope) rightIntercept.append(yintercept) else: None elif slope < -0.3: if x1 < 600: yintercept = y2 - (slope*x2) leftSlope.append(slope) leftIntercept.append(yintercept) #We use slicing operators and np.mean() to find the averages of the 30 previous frames #This makes the lines more stable, and less likely to shift rapidly leftavgSlope = np.mean(leftSlope[-30:]) leftavgIntercept = np.mean(leftIntercept[-30:]) rightavgSlope = np.mean(rightSlope[-30:]) rightavgIntercept = np.mean(rightIntercept[-30:]) #Here we plot the lines and the shape of the lane using the average slope and intercepts try: left_line_x1 = int((0.65*img.shape - leftavgIntercept)/leftavgSlope) left_line_x2 = int((img.shape - leftavgIntercept)/leftavgSlope) right_line_x1 = int((0.65*img.shape - rightavgIntercept)/rightavgSlope) right_line_x2 = int((img.shape - rightavgIntercept)/rightavgSlope) pts = np.array([[left_line_x1, int(0.65*img.shape)],[left_line_x2, int(img.shape)],[right_line_x2, int(img.shape)],[right_line_x1, int(0.65*img.shape)]], np.int32) pts = pts.reshape((-1,1,2)) cv2.fillPoly(img,[pts],(0,0,255)) cv2.line(img, (left_line_x1, int(0.65*img.shape)), (left_line_x2, int(img.shape)), leftColor, 10) cv2.line(img, (right_line_x1, int(0.65*img.shape)), (right_line_x2, int(img.shape)), rightColor, 10) except ValueError: #I keep getting errors for some reason, so I put this here. Idk if the error still persists. pass def hough_lines(img, rho, theta, threshold, min_line_len, max_line_gap): """ `img` should be the output of a Canny transform. """ lines = cv2.HoughLinesP(img, rho, theta, threshold, np.array(), minLineLength=min_line_len, maxLineGap=max_line_gap) line_img = np.zeros((img.shape, img.shape, 3), dtype=np.uint8) draw_lines(line_img, lines) return line_img def linedetect(img): return hough_lines(img, 1, np.pi/180, 10, 20, 100) hough_img = list(map(linedetect, canny_img)) display_images(hough_img)
This should output the following:
You can then add the previously detected lanes to your original image using:
def weightSum(input_set): img = list(input_set) return cv2.addWeighted(img, 1, img, 0.8, 0) result_img = list(map(weightSum, zip(hough_img, imageList))) display_images(result_img)
Which should output the following:
We've detected the lanes somewhat accurately now, but if we were to apply this to a video stream, the lines would oscillate way too much, and is likely to remain unstable. We can make the line more stable by using interframe averaging. Basically, we will be averaging the slope and intercepts of the lane lines from previous frames. This way, the line moves much more slowly and accurately. Luckily, the previous functions already take a weighted average between frames and images.
But how would you apply this to a video? The MoviePy library provides some great tools for video processing. You can easily install moviepy using:
pip install moviepy
Make sure you import libraries into your code before using them. The following function allows you to apply the entire pipeline to video:
def processImage(image): interest = roi(image) filterimg = color_filter(interest) canny = cv2.Canny(grayscale(filterimg), 50, 120) myline = hough_lines(canny, 1, np.pi/180, 10, 20, 5) weighted_img = cv2.addWeighted(myline, 1, image, 0.8, 0) return weighted_img
You can use that function like this:
output1 = 'test_videos_output/solidYellowLeft.mp4' clip1 = VideoFileClip("test_videos/solidYellowLeft.mp4") pclip1 = clip1.fl_image(processImage) #NOTE: this function expects color images!! %time pclip1.write_videofile(output1, audio=False)
And you should see the following video in your output directory:
And that's it! A simple lane detector that works!Limitations
This lane detection system is far from perfect, it was one of my first versions. It can't detect lanes outside of perfect conditions, and can only detect straight lines. I'll be posting a curved lane detector sometime soon. In the meantime, make sure to subscribe to my channel, and follow me on Hackster!