Understanding HTTP Requests & Responses in Golang

How to communicate with API endpoints

HTTP requests are a fundamental part of almost any application, it allows us to retrieve and submit data from endpoints which we can then manipulate. In Golang a simple HTTP GET request can be “not so simple” to newcomers. This article will look to build your understanding of the HTTP request and response in Golang and enable you to confidently make HTTP requests in your Golang application, let’s begin.

A Quick Overview of a Response

Before we start lets quickly go over what a response consists of, for now will just look at a GET request. We’re going to use httpbin.org which is a great resource for playing about with and testing HTTP requests and responses.

Lets run a curl request to the /get endpoint and inspect the output

curl -v https://httpbin.org/get

Below is a shortened version of what is returned, I’ve omitted the parts that won’t be explained today but feel free to investigate them yourself ;

< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Content-Type: application/json
< Date: Tue, 08 Oct 2019 21:14:39 GMT
< Referrer-Policy: no-referrer-when-downgrade
< Server: nginx
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-XSS-Protection: 1; mode=block
< Content-Length: 200
< Connection: keep-alive
<
{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Host": "httpbin.org",
        "User-Agent": "curl/7.54.0"
    },
    "origin": "87.0.213.186, 87.0.213.186",
    "url": "https://httpbin.org/get"
}

In the above response there is a status code, a collection of headers and a response body. The status code is represented by,

< HTTP/1.1 200 OK

The status code is the specifically the “200” in the above. The status code tells us the status of our request, ie whether it failed or was successful, a 200 means our request was successful. The HTTP/1.1 is just the version of HTTP that was used for the request and the OK is the corresponding message for a 200 status code. See here for a full list of status codes.

The headers are a list of key value pairs and are represented by,

< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Content-Type: application/json
< Date: Tue, 08 Oct 2019 21:14:39 GMT
< Referrer-Policy: no-referrer-when-downgrade
< Server: nginx
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-XSS-Protection: 1; mode=block
< Content-Length: 200
< Connection: keep-alive

Each line represents a key value pair and enables the client and the server pass additional information with an HTTP request or response. For example “Content-Type: application/json” lets us know what format we can expect the body to be in so we can parse it accordingly.

Finally we have the response body, this is the data we want to retrieve from the endpoint and is represented by,

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Host": "httpbin.org",
        "User-Agent": "curl/7.54.0"
    },
    "origin": "87.0.213.186, 87.0.213.186",
    "url": "https://httpbin.org/get"
}

Once we have this data we can load it into variables within our program and manipulate it to serve our purpose.

A Simple GET Request

In most programming languages, grabbing some data from an endpoint and loading it into a variable via a simple HTTP GET request requires very little work. Lets use python as an example and take a look at a snippet of code for a simple http get request to an api endpoint.

Simple python script to getting some data from a api endpoint

In this example we import requests, we then make our HTTP GET request to the “https://httpbin.org/get” api endpoint using requests.get("https://httpbin.org/get") and set the result in the variable “response”. We can then access the status code returned using response.status_code, the headers can be accessed using response.headers and finally the body in JSON format loaded into a python dictionary using response.json(), we can get the raw body in string form using response.text. Finally we print out the variables to the console for us to inspect.

Golang is much the same however when you first start it can appear much more complicated. Hopefully you’ll see by the end of this article that Golang follows roughly the same process with a few nuances.

A Simple (Not so Simple) GET Request in Golang

A not so simple http get request to the same api endpoint we saw in the python code above

A lot of the above code is essentially the same as the python code. The import of net/http mirrors the import of requests and the setting of the resp variable to be equal to http.Get() again is extremely similar to request.get(). To understand how HTTP requests and responses work in Golang we must first understand the http.Get() method and its return types.

Golang is a statically typed language so understanding the return types is essential. We can find out return types and the implementation of that type by looking at the documentation. Being able to read documentation is a great skill to have and fortunately Golang provides some of the best docs out there. Follow along and you’ll realise reading the documentation isn’t as hard as you thought it would be.

First lets look at the return type of http.Get(). To find this we just need to navigate to the net/http package and method Get() in the documentation. (See here for the doc https://golang.org/pkg/net/http/#Get)

func Get(url string) (resp *Response, err error)

Here we can see that the method signature for “Get” returns 2 items, a pointer to a Response type (*Response) and an error type. The Get method takes in a single string variable called url.

Quick detour, a pointer is essentially the memory address of a variable so instead of passing around (duplicating) a huge piece of data you can simply pass around the memory address and then read the data at that memory address when you need to. This leads to performance improvements. See here for more information on pointers.

Now that we know the return variable is of type *Response lets find the docs on Response to understand what it consists of (https://golang.org/pkg/net/http/#Response).

The Response Struct, The Key Sections

The Response struct just like python has fields like Headers, Body and StatusCode. Their types however is where the confusion can begin. For now I’m going to simplify the Response struct to just the 3 items below so we can get a grasp of how these key pieces work.

type Response struct {
    StatusCode int
    Header     Header
    Body       io.ReadCloser
}

StatusCode Field - is the simplest, it returns a int just like in python no additional work is required.

Header Field - is slightly more complicated, it is of type Header which means we need to jump a little further down the rabbit hole to understand what a Header type is. Again the docs come to the rescue here (https://golang.org/pkg/net/http/#Header).

type Header map[string][]string

We can see Header is ultimately a map[string][]string. This is a dictionary or hash map where the keys are strings and the contents are arrays of strings, heres an example to clarify.

{
    "key1":["value1.1", "value1.2", "value1.3"],
    "key2":["value2.1", "value2.2", "value2.3"]
}

Body Field - this is where I personally was very confused initially. In python you call request.json() and it takes the body and loads it into a dictionary ready for you to use within your code. Golang requires a few additional steps to get to this stage.

In Go you get back an object of type io.ReadCloser. So what the hell is an io.ReadCloser. An io.ReadCloser is an interface that implements the Reader and Closer interfaces which in turn implement a Read function and a Close function. Still confused? Don’t worry I was too.

Let’s explain the component parts first. A Reader is simply an object that has data inside of it which you can read out of it and an Closer is something that provides a generic way to close streams.

So our resp.Body is an object that has a some data inside of it which we can read out and has a function that allows us to close the stream. Docs here https://golang.org/pkg/io/#Reader, https://golang.org/pkg/io/#Closer.

type Reader interface {
    Read(p []byte) (n int, err error)
}
type Closer interface {
    Close() error
}

We want the body data in a format we understand so first of all we need to read it out. We want to make sure we close this operation when we are done to prevent resource leaks (Hence why we needed the “Closer”). We do this using defer resp.Body.Close() this says close the response body at the end of the function. We can now read out the data using ioutil.ReadAll() which will read the entire contents of the Reader into memory and then returns that data as an array of bytes. To convert the array of bytes to a string we simply use string(bodyBytes). Under the hood the ioutil.ReadAll() function implements a buffer to read the data.

A data buffer (or just buffer) is a region of a physical memory storage used to temporarily store data while it is being moved from one place to another

We now finally have the body in a format we can understand and is equivalent to the “response.text” variable in the python code. Here is the output I got from the Golang code posted above, I’ve added additional spacing and comments to make it easier to read,

// fmt.Println(resp.StatusCode)
200
// fmt.Println(resp.Header)
map[Referrer-Policy:[no-referrer-when-downgrade] Server:[nginx] X-Frame-Options:[DENY] Connection:[keep-alive] Access-Control-Allow-Origin:[*] Content-Type:[application/json] Date:[Sun, 13 Oct 2019 19:51:29 GMT] Access-Control-Allow-Credentials:[true] X-Content-Type-Options:[nosniff] X-Xss-Protection:[1; mode=block]]
//fmt.Println(resp.Header["Content-Type"])
[application/json]
// fmt.Println(resp.Header["Content-Type"][0])
application/json
// fmt.Println(bodyString)
{
  "args": {},
  "headers": {
    "Accept-Encoding": "gzip",
    "Host": "httpbin.org",
    "User-Agent": "Go-http-client/1.1"
  },
  "origin": "87.0.213.186, 87.0.213.186",
  "url": "https://httpbin.org/get"
}

There are a few more steps needed to get the data into the equivalent of a python dictionary I have run through this process in another article here. It provides a run down of how to convert this JSON string into a usable Golang variable. In future articles I will discuss how to set headers for your GET request and run through a HTTP POST request as well.

We are now more or less at the same point as we were in the python code we have access to the status code, the headers and the body albeit in string format. Hopefully you now have an understanding of what the main components of a HTTP response are and how to use them even if you are still a little unsure of why they are implemented in this way. We have used the docs and followed the breadcrumb trail to understand what each component is made of and how it works. This improved understanding will hopefully enable us all to write better, cleaner code.

Thanks again for reading, as always any feedback is welcome and please let me know if I’ve made any mistakes.