My opinion after testing some AI code assistant

AWS CodeWhisperer

AWS CodeWhisperer is AWS take on the subject. It's tied to your AWS account and AWS extension for your editor. It's free for individuals.

According to my tests, it's the worst overall. On the bright side, it did generate a few test cases that were more relevant to my use case than other tools and can handle imports automatically. But it suggested any way too often in its TS completions, proposed less variations between suggestions, proposed more irrelevant suggestions, proposed more complex suggestions and relied too often on line to line suggestion (versus block/function suggestion for other assistants). It's also way slower and felt unresponsive at times.

It's also the only tool that proposes a security scanner. I tried in on a piece of Python code and its suggestions were good: I was lacking a context manager and doing a SQL injection which the tool spotted. It was very slow, even on my small file (~10 lines of code). I don't know how it behaves nor what it reports on larger and more complex pieces of code.

To conclude: It's definitely a pass for me. In this current state, it's also a pass if you are tied to AWS product. Just code without, you should have a better time. The security scanner can be a good thing if it works properly on large code bases though.

GitHub Copilot

GitHub Copilot is you guessed it made by GitHub. It's linked to your GitHub account and requires a dedicated extension in your editor. It's not free even for individuals, but comes with a 1 month trial period (that's what I used).

According to my tests, it proposed the best suggestions of all tested tools. It's even good at not messing up too much parentheses and curly braces completions in existing code (where all other tools had some/more issues). It's also fast at responding. It does the job and does it well (for an IA assistant).

There is also a chat in beta. I couldn't try it: it's only opened for business accounts right now. But, from my experience with codeium it should be a valuable additions.

Codeium

Codeium is a product made by a startup. It requires an account (you can use your Google account) and a dedicated extension in your editor. The VSCode extension is the most complete since it supports all features (completion and chat), but I tested it in PyCharm and code completion work fine (I lacked the chat though). You can test it in their playground before creating an account nor installing anything. That's a good thing in my opinion. It's free for individual use. It's also the tool I used beyond my small IA assistants tests (because it's good, the chat is quite handy and it's free).

It's suggestions aren't always the most relevant and it tends to fail for parentheses/curly braces completions. When generating test cases, I had to guide it to have something good. It invented a method named toBeUpperCase and in a forEach loop instead of using the item it used the array and the proper index. Weird (but working). Inventions struck me more with this tool than with the others (but it's also the one I used most, so I may be biased). Its different proposals are good and different. By default, in TypeScript, it create fat arrow functions on multiple lines even when the result could be a one liner. In Python, it tends to propose pass as the body of the function by default (you can then make it write it).

It's also the only assistant I tested that propose more features than mere code completions:

• It comes with a chat that is a must use. Instead of searching Stack Overflow for something, most of the time I could just ask it how to do something (like a form in Angular) and it would respond. Depending on the answers, I could get a code snippet I could use, a detailed explanation or I had to ask it again (and sometimes go to Stack Overflow).
• It proposes tools to refactor code: you select a piece of code, run a default refactoring action or explain what refactoring you want. Then it proposes a code snippet you can apply to take the refactoring into account. I only used it on simple tasks, but it worked well. This sounds really promising since refactoring can be really boring.

How about learning?

You can use these assistants to help you write code in a language you don't really know. I tried that with Elm. It was more pleasing that doing it only with the documentation and web searches: you can rely on completions to get to your goal more rapidly and with less frustration. Especially with a language as different from the ones I know as Elm.

Did I really learn or practice Elm? No! That's not really a problem in this context, because I didn't really want to learn it. I think it can be if I really tried: how could I get into the right state of mind, learn by practising if a tool gives me the answers? I could understand the code and edit it a bit thanks to my experience, but overall, I don't really know if the code is actually good.

More generally, I don't think you can truly learn anything with a tool like this: you need to think, fail and write code by your own to learn. I don't think there is an alternative to this. But these tools will make it harder to do it correctly. For instance, I tried to do some type challenges in TS. With the autocomplete on, I just got the answers. Answers I could not have found on my own without trying a bunch of things (and some I just couldn't get without the solutions since they relied on things I didn't know).

And you also need to know whether the code is good, secure and maintainable.

The chat can be useful to get relevant information right from your editor though. But I think you'll still need to write code by yourself, experience developers to review your code and blog/courses/books to move forward in your programming journey. So I don't advise using completions when trying to learn: you could feel more productive at the start, but I think it will come and bite you in the long run.

How about using it on the job?

With all that said, these assistants sound like the prefect tool to help to you in your job. There are some points to consider before that though:

• Technically, you must pay for them to use them in a professional context. They are not expensive, but depending on your company size, it can become quite expensive.
• They are many legal considerations to take into account:
  • You could violate copyrights: the assistants learned from OSS projects and may generate license protected code that will end up in your project. It's mostly dangerous for GPL protected code. While technically you could do it before, you had to conscientiously do it, search for it and use it. Now, it's at the finger tip of millions of developers that will have no idea they are infringing any copyright.
  • Who owns the output of an IA tool is not clear.
  • This requires to send code to a distant server you don't control. This could result in code/data/secret leakage. You need to make sure your provider won't learn from your code. You need to trust it and read its data policy: to be more accurate, it's too important to just accept without understanding it in a professional context and you need to get legal involved.

So don't do this on your own, involve your co-workers and management. Even to use the tool from time to time at your job with a personal account. To avoid having problems over this, you must get management support before using the tool. Otherwise, this too can come and bite you!

Wrapping up

• AI assistant can be very powerful tools to help you refactor code, get answers faster and help you write code (either tricky or boring).
• Probably not the best tool to learn and practice programming. I think it applies to all other activities.
• You still need to rely on your experience to maintain code quality and security (and to make the whole thing work).
• Make sure to have management support if you want to use them at work: they bring a lot of non technical issues you cannot solve on your own.

Other takes on this:

• Exploring Generative AI.