When running a socket server and client in python, an abrupt disconnect by the client is handled differently by the server when running both locally versus running both in docker containers and through a docker network. If it matters, all of these tests are done on a Linux machine.
Here is a simple socket server that serves incrementing bytes:
import socket
import time
def main():
# Create a TCP/IP socket
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(("0.0.0.0", 12345))
server_socket.listen(1)
print('Server bound and listening.')
try:
# Wait for a connection
print('Waiting for a connection...')
connection, client_address = server_socket.accept()
print('Connection established from:', client_address)
try:
# Serve incrementing bytes
current_byte = 0
while True:
connection.sendall(bytes([current_byte]))
print(f'Sent byte: {current_byte}')
current_byte = (current_byte + 1) % 256
time.sleep(1)
finally:
connection.close()
print('Connection closed.')
except KeyboardInterrupt:
print('Server interrupted by user, shutting down.')
finally:
# Clean up the server socket
server_socket.close()
print('Server socket closed.')
if __name__ == '__main__':
main()
And here is the corresponding client to consume the incrementing bytes:
import socket
import time
import os
# Override when running in docker to point to the correct host.
SERVER_HOST = os.getenv("SERVER_HOST", "localhost")
def main():
# Wait for server to come up.
time.sleep(1)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((SERVER_HOST, 12345))
while True:
bytes_received = sock.recv(1)
print(f"Received bytes: {str(bytes_received)}")
if __name__ == "__main__":
main()
When I run these scripts locally (no docker containers and loopback), a BrokenPipeError
is raised by connection.sendall(...)
when I either Ctrl+C or kill the client.
Server:
Server bound and listening.
Waiting for a connection...
Connection established from: ('127.0.0.1', 44778)
Sent byte: 0
Sent byte: 1
Sent byte: 2
Sent byte: 3
Connection closed.
Server socket closed.
Traceback (most recent call last):
File "/path/to/socket_server.py", line 43, in <module>
main()
File "/path/to/socket_server.py", line 23, in main
connection.sendall(bytes([current_byte]))
BrokenPipeError: [Errno 32] Broken pipe
Client (interrupted with a user-input Ctrl+C):
Received bytes: b'x00'
Received bytes: b'x01'
Received bytes: b'x02'
^CTraceback (most recent call last):
File "/path/to/socket_client.py", line 21, in <module>
main()
File "/path/to/socket_client.py", line 17, in main
bytes_received = sock.recv(1)
^^^^^^^^^^^^
KeyboardInterrupt
However, when running in docker containers and over a docker network, the server never detects that the client disconnected. Here is a docker-compose file to spin up the services and connect them together:
version: "3.9"
networks:
socket-net:
name: socket-net
external: false
services:
client:
image: python:3.11
volumes:
- ./scripts:/scripts
environment:
# Show prints without explicit flush
PYTHONUNBUFFERED: 1
SERVER_HOST: "server"
command: ["python", "/scripts/socket_client.py"]
networks:
- socket-net
server:
hostname: server
image: python:3.11
volumes:
- ./scripts:/scripts
environment:
# Show prints without explicit flush
PYTHONUNBUFFERED: 1
command: ["python", "/scripts/socket_server.py"]
networks:
- socket-net
When I interrupt the socket_client
with a docker kill
, the server doesn’t always raise an exception:
ebenevedes@machine:/path/to/socket_mre$ docker compose up --force-recreate
[+] Running 3/0
✔ Network socket-net Created 0.0s
✔ Container socket_mre-client-1 Created 0.0s
✔ Container socket_mre-server-1 Created 0.0s
Attaching to client-1, server-1
server-1 | Server bound and listening.
server-1 | Waiting for a connection...
server-1 | Connection established from: ('172.25.18.2', 54428)
server-1 | Sent byte: 0
client-1 | Received bytes: b'x00'
client-1 | Received bytes: b'x01'
server-1 | Sent byte: 1
client-1 | Received bytes: b'x02'
server-1 | Sent byte: 2
client-1 | Received bytes: b'x03'
server-1 | Sent byte: 3
client-1 exited with code 137 # Killed with `docker kill socket_mre-client-1` from a different terminal.
server-1 | Sent byte: 4
server-1 | Sent byte: 5
server-1 | Sent byte: 6
server-1 | Sent byte: 7
server-1 | Sent byte: 8
... # Server continues to send bytes without raising an Exception
- Why do sockets exhibit different behavior when connecting through loopback versus through a docker network?
- How can I consistently detect a disconnecting client from the server, no matter what the network path is between the server and client? I have seen a similar issue with EKS K8s deployments, but don’t have a minimum reproducible example for that case.
2
Answers
Try this for your client:
My first answer doesn’t really solve the issue. A more useful answer is to have the server listen for a response.
client:
server: