GitHub Co-Pilot Generating Different Outputs in Different Editors

I’ve not been able to write something new lately, in fact, I think this is the first post of the year. So happy new year 2024, insert :tada: emoji. So I’ve been pretty much busy with work, and travelling, networking at conferences, the usual. At the start of 2023, I had never left Sri Lanka in my whole life, but by the end of 2023, I’d visited 5 different countries, including the US.

But back to the topic, when my friend Patrick showed me that he has been using GitHub co-pilot, I had the usual scepticism that I’ve always shown, that I don’t want to switch to co-pilot, it will take away your creative thinking/problem-solving capacity as opposed to you thinking and solving a problem.

This is also why I don’t want ChatGPT to write content on letters on my behalf because I learned the ability to creatively write something after years of practice, and I just don’t feel like giving it away or losing that by giving it to an AI to do.

Zed editor (zed.dev), which comes with co-pilot and ChatGPT built in. So since I’m always up for trying new editors, I installed it on my MacBook Air, and since it has co-pilot built in, I decided to bite the bullet and get a Co-Pilot subscription.

As for 2024, I started a simple project of creating a bookmark manager for myself, even though it goes against my previous post “Ideas that have been beaten to death”, and started coding with it, since I’m the only coder and I just wanted to ship it quickly (or just use it for myself).

And now that I’ve been using co-pilot for the last 2 weeks, wow, I feel I missed a lot last year. Also I don’t think I will go back to the non-co-pilot days. It’s like discovering typescript, once you get to know typescript, you won’t go back to using vanilla JavaScript again. Your life becomes so much easier, and it’s the same feeling I get when using co-pilot.

I still don’t give the co-pilot the power to decide the logic over my code, but it’s pretty neat when you already have the logic in your head, to auto-complete the lines of code for you, and you can decide to override the co-pilot suggestion, and write your own code at the same type. I think that’s a compromise that I can make, rather than promoting co-pilot to solve and come up with the code for you.

And what I now use co-pilot to generate documentation for me, come on we all know that as coders we tend to like writing code over writing documentation. And I’d like to give co-pilot the chance to write the documentation for me.

So I’ve been working on 2 laptops, one a ThinkPad and MacBook Air, on my ThinkPad, I use VsCodium, and Zed on my MacBook. So there was some documentation that was inside my ThinkPad which I’ve not pushed to the repository. Forgetting this, I generated documentation for the same file on Zed and pushed it to the repository, which triggered a merge conflict when trying to merge the code in the ThinkPad.

It surprised me that for the same function, the co-pilot has generated two different documentation, with different quality and different sizes. I have a very basic knowledge of LLMs, and I don’t know how co-pilot generates content. But it’s fascinating that the co-pilot generates different output for the same input, in two different editors running on two different devices.

And it’s not just one function, but it was a recurring thing throughout the file.

I am not going to say that co-pilot generates better documentation for VsCode based VsCodium over Zed, or maybe Zed is sending a different request to the co-pilot servers than VsCodium resulting in a different output. But it is something interesting to see, as you expect to see the same quality content irrespective of the editor and only depending on the input.

I’d love to hear some thoughts, and theories on why co-pilot has this behavior, or is this something related to AI? And has anyone else seen similar behaviour?

Even though I like using co-pilot, I still feel some moral dilemma in using copilot, trained by open source code, Microsoft using it for commercial purposes, and me using a product that is trained by open source projects without me giving in return. But overall I still feel it’s a good product, especially when you are coding everything by yourself, and doing things like documentation which you might skip.