Github Copilot Technical Preview First Impressions

July 02, 2021

I’m not sure what the methods are for giving access to the Github Copilot technical preview, but I was lucky enough to get accepted to the preview today. Here are my initial thoughts:

I thought we were several years out from something like this

The future is now gif

I’m no AI expert but I have followed the field fairly closely for many years. I have always said that software engineering jobs were safe, and that AI would not be a threat anytime soon. I still think that’s the case for many jobs, but not all of them. This model is quite good, and it’s the first one I’ve seen in the wild that seems likely to change the way I write software. As deep learning continues to improve and models continue to improve into the future, some of the more remedial or entry-level jobs will likely be affected.

Legal complications will be interesting to watch

man in suit saying outrageous, egregious, preposterous I hope that legally I'm able to use this for my day job. I think it would increase my productivity a fair bit. Microsoft is a big company, and compliance is strict and always top of mind. I know it's a product created by a Microsoft-owned company, but I wouldn't be shocked if we're not allowed to use it internally.

It’s not even close to perfect, but it will be useful

Marge Simpson saying 'Well, he's not perfect, but he is very--' From my limited experience, Copilot gets you off on the right foot but will leave gaps and details out that you'll need to fill in. Sometimes it spits out things that you don't actually need, such as importing random libraries that are likely often correlated with similar tasks. More often it leaves things out that you need. To do anything more than barebones basics, you'll still need to know how to fill in the gaps.

As a fun example, as I was writing this blog post in Markdown, I pressed enter after the heading and it created an image tag with a broken giphy link and randomly generated alt text stating "A Robot arm holding a block of text":

Suggested autocomplete in VSCode for a broken giphy link (Sorry for the small text in the image!)

It was still faster for me to insert the image than it would have been, I just had to understand what needed to change about it to make it valid. I liked this as it seemed it was learning the pattern in the current file and suggesting things that would make sense. Very impressive.

It crushes common leetcode style problems

Before I had access to the preview I saw this from one of my favorite twitter accounts @davidfowl:

need a palindrome? it’s got you covered. Random number generators? yep. Reverse characters in a string? you betcha. And the solutions are often very efficient and well written.

I typed in the comment in the below snippet and the pressed tab twice to get the following:

//https://leetcode.com/problems/linked-list-cycle/submissions/
const hasCycle = (head) => {
    if (!head || !head.next) {
        return false;
    }
    let slow = head;
    let fast = head;
    while (fast && fast.next) {
        slow = slow.next;
        fast = fast.next.next;
        if (slow === fast) {
            return true;
        }
    }
    return false;
}

I then copied and pasted the whole thing into Leetcode. The code ran perfectly and was faster than 90.03% of JavaScript online submissions for Linked List Cycle.

Leetcode Results

I tried this with three other common leetcode problems and got similar results.

The AI has obviously been trained on many, many attempts at these kinds of problems. If it’s a paid service and you have take-home interview problems, applicants should absolutely pay for the service to compare your results with the AI’s results. Although I’m sure companies will work to mitigate that kind of behavior in the future, it can also be used as a learning tool.

Conclusion

The more I use Copilot the more impressed I am by it. I’ve found that many of the little syntaxy things that I would usually look up on StackOverflow, Github, or Google are now available to me in the form of autocomplete suggestions. I’m also finding that I can write much faster and more efficient. I’m not sure if this is a good thing or a bad thing, but I’m going to keep using it. I also didn’t write the last two sentences myself. Not one word of them. 🤷‍♂️


Profile picture

Hi, I'm Brock. I'm a Software Engineer at Microsoft. You should come learn with me. Oh, and you should also follow me on Twitter. 😊

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer’s view in any way.

© 2021, Brock Taylor. Built with Gatsby. Powered by Azure.