Apr 21, 2019
I've been working on a tool that requires a way to reset the configuration of a local
development Envoy. The Envoy admin's /quitquitquit
endpoint combined with the
docker-compose
restart: always
parameter does the trick but it's quite slow.
Fourtunately Envoy has a "hot restart" feature that can reset an Envoy's configuration
without the slow container restart.
We start Envoy using the provided
envoy/restarter/hot-restarter.py
script which will restart Envoy when it receives a SIGHUP
.
hot-restarter.py start-envoy.sh
Don't forget the include the $RESTART_EPOCH
flag in your start-envoy.sh
script.
--restart-epoch $RESTART_EPOCH
We can then send a SIGHUP
signal to the container which will trigger a hot restart.
$ docker kill --signal=SIGHUP <your-proxy-container>
And now we can check the restart_epoch
before and after the SIGHUP
to see if Envoy has
restarted.
$ curl -s localhost:9901/server_info | grep restart_epoch
"restart_epoch": 0,
$ docker kill --signal=SIGHUP <your-proxy-container>
$ curl -s localhost:9901/server_info | grep restart_epoch
"restart_epoch": 1,
My tool doesn't use curl
to query Envoy's restart_epoch
. Instead, I use Go's http
package and a bit of protobuf parsing. Something along the lines of this:
client := &http.Client{
Timeout: time.Second,
}
resp, err := client.Get("http://127.0.0.1:9901/server_info")
if err != nil {
return nil, err
}
defer resp.Body.Close()
var serverInfo v2alpha.ServerInfo
err = jsonpb.Unmarshal(resp.Body, &serverInfo)
return &serverInfo, err
The mystery is that if I use the above code to query the restart epoch before and after a
hot restart, the client takes an absurdly long time to show the updated restart epoch.
(Think 20 seconds.) I can curl
for the restart epoch and get the bumped epoch version
while my Go client is still returning the old one.
I suspect this is some issue where my Go client is still connected to the old Envoy rather
than the new one. Not sure how to fix this yet. Currently I'm shelling out to curl
. I'll
update this post when I've found a less hacky solution.