My wife and I recently received access to Tesla’s Full Self-Driving (FSD) Beta program.
Over the past month, we’ve learned a lot about the current state of self-driving cars from our first-hand experience using FSD for every trip we’ve taken so far this year in our 2021 Tesla Model S.
In addition, with my background in artificial intelligence, machine learning, and data science, I have a unique understanding of what’s actually happening “under the hood” as our car is making its driving decisions.
So, for those of you interested in learning more about the current state of self-driving cars or what it’s like to teach an AI to drive itself, I thought I would document some of my key observations over this past month.
When we recieved our Tesla Model S in October 2021, there were roughly 150,000 Tesla owners that had requested to be part of the FSD Beta program. However, there were only 2,000 cars with high enough safety scores to be admitted into the program. This means that only the top 1-2% of drivers get admitted.
We requested access to the FSD program as soon as the option became available on our vehicle. Then, we had to drive very safely over the next month. The system deducted points for hard braking, aggressive turning, unsafe following, taking our hands off the wheel, and forward collision warnings.
The more you drive the car in Autopilot though, the easier it is to get a high safety score. So, we drove on Autopilot as much as possible and drove like a grandma the rest of the time. We maintained a safety score of 98% or greater for the duration of our time waiting for access to the FSD Beta program.
On Christmas Day, we recieved a notification from Tesla stating that we had been granted access to the FSD Beta program. We immediately installed the update and took the car for a test drive at our first opportunity. We recieved FSD Beta v10.8 and have used it for the duration of our experience thus far.
Since we had already driven our car on Autopilot for most of our miles since we recieved our Model S in October 2021, we were already very well versed in when to trust the car to drive itself and when to take control. This helped tremendously with safely monitoring the car’s behavior in FSD mode.
In addition, with my background in AI\ML\DS, I had a pretty good idea of what situations might give the car issues with it’s perception, planning, and/or execution. This combination of knowlege and experience made both my wife and I feel comfortable with allowing the car to drive itself in FSD mode.
I was previously under the impression that Tesla used a single software stack for driving on Autopilot and a separate stack for driving on both highways and streets with FSD Beta. So, I was suprised to discover that driving on highways with FSD Beta is identical to driving on highways with Autopilot.
Essentially, when you’re driving on highways with FSD, the car uses the same Autopilot features that are available to everyone who owns a Tesla capable of Autopilot. As soon as you get to your off-ramp though, the vehicle switches into it’s FSD software stack. However, this will change in FSD Beta v11.0
Autopilot on highways and interstates is already pretty rock-solid. It provides traffic-aware cruise control, autosteer, auto lane change, auto navigation, and semi-automatic handling of traffic lights and stop signs. This is all in addition to the standard safety features like collision avoidance and lane assist.
It still has issues in some scenarios though. It drives too slow on some stretches of road, it misses certain off-ramps, it doesn’t properly use HOV lanes, and it’s not a very most polite driver. So you often have to “nudge” it with your accelerator, speed control, and turn signals or take over from time to time.
Driving on streets is where the FSD Beta features become important. FSD can navigate city streets just like it does highways with Autopilot. However, it still has diffiulty on some streets with road construction, atypical lane designs, or that lack clear lane markings or edges.
It can also automatically handle intersections with street lights, stop signs or no traffic control at all. It follows all the usual traffic rules at metered intersections. At unmetered intersections, if it has limited visibility, it creeps ahead until it is confident in its ability to proceed safely.
It’s definitely not as well polished while driving on streets as it is on highways yet. We’ve had a few incidents with abrupt stopping — though we can never quite figure out why. In addition, one unfinished road near our house consistantly give it problems of all kinds.
To help it make the right decisions, you have to consistently “nudge” it using your accelerator, brake, and turn signals. If it’s making a significant error, then you need to take control altogether. It still unknown to me whether Tesla is collecting training data on these nudges or just the disengagements.
FSD Beta gives you access to the Smart Summon feature. When your car is parked, it allows you to request the car to come pick you up.
There are currently two versions called Summon and Smart Summon. Summon just goes forward or backward to pick you up whereas Smart Summon can navigate complex environments.
So far, we haven’t had much luck with the Summon feature. We can get it to back out of our driveway about 40 feet. However, we can’t get it to do much more than that. I’m not sure if there’s an issue that’s preventing the “Smart” portion of the summon feature to work. However, it’s currently not very useful.
FSD Beta also gives you access to the Autopark feature. It allows the car to detect an available parking space and then back the car into that space automatically.
Essentially, you just drive through the parking lot until you find a free space. Then you drive slowly past the open space and a little blue “Autopark” button shows up. Hit the button and the car does a 2-point maneuver to back itself into the space. It’s perfectly centered between the parking space lines every time.
When this feature works, it is awesome. However, it rarely seems to detect parking spots no matter how close, how far, or how slow we drive past them.
The most important thing you need to know about driving with FSD Beta is that it’s essentially like teaching a student driver how to drive a car. One moment it’s driving so well that you think “wow… how is it even doing that!” and then the next moment it will make the stupidest decision you’ve ever seen.
So, you need to constantly monitor its behavior and be ready to take control in a moment’s notice when it makes a mistake. Each time you take control to correct a mistake, this is sending a signal to Tesla’s engineers (via auto-labeling) to inspect the data and use it to train future versions of the software.
In essence, by driving a Tesla with FSD and correcting its mistakes you are teaching the car how to drive itself. However, this reward/punishment feedback loop is a lot longer than a human driver. It takes a few weeks for new versions of FSD to be released with your training data integrated into the models.
Ultimately, I’m very happy to know that I’m playing an active role in teaching the next generation of AI how to drive. In addition, I’m very much looking forward to the significant improvements that will be coming in FSD Beta v11 in the next month or two.
If you want to learn how self-driving AI works, I recommend you watch the presentation from Tesla’s engineers at their AI Day event.
If you want to learn more about the future of AI (in general) and how it will impact you, your career, and our world, I recommend you watch my free online course on Preparing Your Career for AI.