A few weeks ago I ran My standard set of programming tests against the free version of the Perplexity.ai Chatbot. In The end of the articleI offered to test against him Pro Version $20/month If there were enough interested parties, I got some applications, so that’s what we’re doing here.
Also: Want a free year of Perplexity Pro? Here’s how to get it
Like most other pro versions, to use Perplexity Pro you need to create an account. You can log in using Google or Apple authentication methods or a SAML login. You can also create an account using your email address, which is what I did.
Unfortunately, the site does not seem to provide you with any way to set a password or any way to multi-factor authenticationThey send you an email with a code and that’s it. I don’t mind receiving a code via email, but I’m very annoyed by web applications that rely solely on an email code without at least a password. But that’s what Perplexity.AI is doing.
Also: 5 Reasons Why I Prefer Perplexity Over Any Other AI Chatbot
The other interesting aspect of Perplexity Pro is its cornucopia of AI models. As you can see in the image below, you can choose from several different models, depending on the type of job you have. I chose Default to see what it did with the tests. After running the tests, I asked Perplexity Pro which model it used for them, and it told me ChatGPT GPT-4.
And with that, let’s do some testing.
1. How to Write a WordPress Plugin
This challenge is a fairly straightforward programming task for anyone with a modicum of web programming experience. It presents a user interface in the admin panel with two fields: one is a list of names to be randomized, and the other is the result.
The only real problem is that the list of names may have duplicates, and instead of removing the extra names, your instructions are to make sure the duplicate names are separated from each other.
Also: How I Test an AI Chatbot’s Coding Ability (and You Can Too)
This was a real feature that my wife needed to use for her e-commerce website. Every month, they spin the wheel and some people qualify for multiple entries.
Using Perplexity Pro’s default template, the AI was able to generate a viable user interface and working code, providing a PHP block and a JavaScript block to control the text areas and randomization logic.
Below are the aggregate results of this and previous tests:
- Perplexity Pro: Interface: good, functionality: good
- Perplexity: Interface: good, functionality: good
- Claude Sonnet 3.5: Interface: good, functionality: failure
- ChatGPT using GPT-4o: Interface: good, functionality: good
- Microsoft Copilot: Interface: adequate, functionality: failure
- Meta IA: Interface: adequate, functionality: failure
- Meta code Call: Complete failure
- Google Gemini Advanced: Interface: good, functionality: failure
- ChatGPT using GPT-4: Interface: good, functionality: good
- ChatGPT using GPT-3.5: Interface: good, functionality: good
2. Rewriting a string function
In each test, I open a new session with the AI. In this test, I ask the AI to rewrite a block of code that had an error. The code was designed to validate input of dollars and cents, which must include a certain number of digits before the decimal point, a possible decimal point, and two digits after the decimal point.
Also: Oops! Microsoft Copilot failed any of my coding tests
Unfortunately, the code I submitted only allowed integers. After a couple of user reports, I decided to send the code to the AI to rewrite. My code uses regular expressions, which are a formulaic way of specifying a format. Regular expressions themselves are fun, but debugging them is not.
In the case of this test, Perplexity Pro did a good job. The resulting validation code correctly flagged items that did not conform to the dollars and cents format, allowing up to two digits after the decimal.
Below are the aggregate results of this and previous tests:
- Perplexity Pro: Successful
- Perplexity: Successful
- Claude Sonnet 3.5: Failed
- ChatGPT using GPT-4o: Successful
- Microsoft Copilot: Failed
- Meta IA: Failed
- Meta code Call: Successful
- Google Gemini Advanced: Failed
- ChatGPT using GPT-4: Successful
- ChatGPT using GPT-3.5: Successful
3. Find an annoying bug
This test had me stumped for a few hours. Before it was a test, it was a bug in the code of a real product. The problem was that what was going wrong was not related to any obvious logic or language problem.
Also: I asked ChatGPT to write a WordPress plugin I needed. He did it in less than 5 minutes
Since I was very frustrated, I decided to send ChatGPT the code and the error dump and ask for help. Fortunately, he found what I had done wrong and gave me guidance on what to fix.
The reason I included this in the test suite is because the bug was not in the language or logic, but in the knowledge of the WordPress framework. While WordPress is popular, knowledge of the WordPress framework is generally considered part of the folklore of a programming environment, something that is passed down from developer to developer, rather than something that would be rigorously learned through a knowledge base.
However, ChatGPT, as well as Perplexity and now Perplexity Pro, found the problem. The bug was due to a parameter calling issue hidden in the framework itself. The obvious answer, which could be reached by simply reading the error messages generated by the code, was actually wrong.
Also: Uber One subscribers get a free year of Perplexity Pro. Here’s how to claim it
To solve this, AI had to demonstrate a deeper understanding of how all the systems work together, something Perplexity Pro successfully accomplished.
Below are the aggregate results of this and previous tests:
- Perplexity: Successful
- Perplexity Pro: Successful
- Claude Sonnet 3.5: Successful
- ChatGPT using GPT-4o: Successful
- Microsoft Copilot: Failed
- Meta IA: Successful
- Meta code Call: Failed
- Google Gemini Advanced: Failed
- ChatGPT using GPT-4: Successful
- ChatGPT using GPT-3.5: Successful
4. Write a script
Well, this is interesting. Perplexity Pro passed this test, but the Perplexity free version failed When I tried it a couple of weeks ago, so, great!
But let’s dig a little deeper into this. The challenge here is that I’m asking the AI to write a script that intersects three environments: Chrome’s DOM (document object model), AppleScript (Apple’s native scripting language), and Keyboard Maestro (a very cool Mac automation tool that’s fairly unknown, but to me, mission-critical).
Most of the AIs failed because they didn’t have any information about Keyboard Maestro in their knowledge bases and as such didn’t provide the code needed for the script to do what I wanted.
Also: How to use ChatGPT to write code: What it can and can’t do for you
Only Advanced Gemini and ChatGPT which uses GPT-4 and GPT-4o passed this test so far. When answering the question, Perplexity Pro provided a Pro search view. As you can see, the Pro search view performed a search for “Keyboard Maestro AppleScript Google Chrome Tabs”. It also used The main Keyboard Maestro forum As a source, what is the best source to get help with Keyboard Maestro coding.
The result was a success.
Below are the aggregate results of this and previous tests:
- Perplexity Pro: Successful
- Perplexity: Failed
- Claude Sonnet 3.5: Failed
- ChatGPT using GPT-4o: It was successful, but with reservations
- Microsoft Copilot: Failed
- Meta IA: Failed
- Meta code Call: Failed
- Google Gemini Advanced: Successful
- ChatGPT using GPT-4: Successful
- ChatGPT using GPT-3.5: Failed
Total Results
Here are the overall results of the four tests:
As you can see, Perplexity Pro only outperforms ChatGPT with GPT-4 and GPT-4o, scoring a perfect 4 out of 4. After running my tests, I checked with Perplexity Pro’s AI and it reported that it used GPT-4 to analyze and respond to my tests.
Considering that GPT-4/4o is the only AI that has passed all four of the tests I have run previously, this makes sense. So far, I have not found any other model that can completely and correctly pass all four programming tests.
Also: How to run dozens of AI models on your Mac or PC, without needing a third-party cloud
If you choose Perplexity Pro, I can say with some confidence that it will be able to do a good job of helping you program.
Have you tried programming with Perplexity, Copilot, Meta AI, Gemini or ChatGPT? What has been your experience? Let us know in the comments below.
You can follow daily updates of my projects on social media. Make sure to subscribe to My weekly update newsletterand follow me on Twitter/X on @DavidGewirtzon Facebook at Facebook.com/DavidGewirtzon Instagram at Instagram.com/DavidGewirtzand on YouTube at YouTube.com/DavidGewirtzTV.