Using GPT-3 to write code

Writing code with ChatGPT.

In the aftermath of ChatGPT’s debut, the Internet is abuzz with GPT’s ability to write code. I used it to write code yesterday, and this is what I thought.

I wanted to create a command line program to interact with APIs from OpenAI and AI21. The design is a single program with subcommands for each company’s APIs:

Usage: /bin/foo_to_be_named <command>

A tool to interact with various LLM models from the CLI

Flags:
  -h, --help                                     Show context-sensitive help.
      --conf-file="llmcli.conf"    file with api keys, created with http://localhost:8080/v1/p/tools

Commands:
  openai create-completion
  openai create-edit
  ai21 complete

Run "/bin/foo_to_be_named <command> --help" for more information on a command.

I like to use Kong as the framework for command line parsing and I wanted to know how to use its subcommand features. Being too lazy to read the documentation, I asked ChatGPT to write me a short program using sub-commands. This is the example it came back with:

package main

import (
	"fmt"

	"github.com/alecthomas/kong"
)

type command struct {
	Verbose bool   `kong:"help='Verbose mode'"`
	Command string `kong:"arg,help='Command to run'"`
}

func main() {
	var cli command
	ctx := kong.Parse(&cli, kong.UsageOnError())

	// Subcommand "run"
	if cli.Command == "run" {
		var runCmd struct{}
		kong.Parse(&runCmd, ctx)

		fmt.Println("Running command")
		if cli.Verbose {
			fmt.Println("Verbose mode is on")
		}
	}

	// Subcommand "list"
	if cli.Command == "list" {
		var listCmd struct{}
		kong.Parse(&listCmd, ctx)

		fmt.Println("Listing items")
		if cli.Verbose {
			fmt.Println("Verbose mode is on")
		}
	}
}

It… works? I guess? But it’s not very good code. In fact, it doesn’t even use the example from Kong’s documentation:

The code that ChatGPT generates will require updates to the main method with every new subcommand. As the number and complexity of the subcommands grow, the code will become unwieldly and error prone. It is not the kind of code I’d use. On the one hand, I can see how GPT-3 can help people new to programming get off the ground. On the other hand, it doesn’t generate good code. In fact, it generates precisely the kind of code I hate seeing in code reviews: The code that works but is unmaintainable.

How would I rate ChatGPT’s ability to write code so far? It failed on the first try. But maybe I just got unlucky.