We ran an experiment to see how easy it is to cheat with ChatGPT in interviews

Overview
We conducted a controlled experiment where interview candidates were allowed to use ChatGPT during live technical interviews without the interviewer's knowledge. The study measured success rates across different question types and examined how easily AI can be used to cheat.

Key Takeaways

• Candidates using ChatGPT performed significantly better on algorithmic coding questions compared to those who did not.
• The advantage varied by question type, with data-structure and system-design questions showing smaller gains.
• Interviewers can detect AI-generated answers by looking for overly polished code or inconsistent reasoning.
• Implementing real-time monitoring or asking follow-up questions reduces the risk of cheating.

Who Would Benefit

• Engineering managers designing interview processes.
• Technical recruiters looking to safeguard assessment integrity.
• Team leads training developers for interview preparation.
• Anyone interested in the impact of generative AI on hiring.

Frameworks and Methodologies

• Controlled A/B testing with blind interviewers.
• Statistical analysis of pass rates by question category.