Portless Ports: Demystifying Kubernetes Port Forwarding

Arjun C -

Introduction

Recently, I was exploring Hexmos' infrastructure, trying to learn all the steps involved in loading up the Feedback app in the browser. During my investigation of a packet's path to the server and analysis of the Nginx configuration file that governs how the web server handles incoming requests, I discovered that the requests were being redirected to http://localhost:32264/. So, I did a 

curl http://localhost:32264

This is the response of the homepage in feedbackweb. When I checked for the port in netstat

 netstat -tuln | grep 32264

the port 32264 was not listed in it.

Why Kubernetes Service Ports Aren't Listed with Others

When I conducted further research, I discovered that the ports of services running within a container are not displayed in netstat, and the karmaweb service (feedbackweb) was operating in a Kubernetes environment. In Kubernetes, service ports are managed differently from host ports. When a service is exposed Kubernetes will assign a port number to it which is referred to as a "service port". These ports won’t be listed with other ports because they are hidden by the Kubernetes network layer. To get more context we can use kubectl is a command line to interact with Kubernetes. The following command will list out the karmaweb service 

kubectl get rc, services | grep karmaweb

Kubernetes Basics

Overview

Kubernetes is an open-source orchestration tool developed by Google for managing microservices or containerized applications across a distributed cluster of nodes. Kubernetes provides a highly resilient infrastructure with zero downtime deployment capabilities, automatic rollback, scaling, and self-healing of containers (which consists of auto-placement, auto-restart, auto-replication, and scaling of containers based on CPU usage).

Reason for using Kubernetes

  • Scalability: It is easy to scale the applications using Kubernetes. It can automatically adjust the number of containers to handle increased traffic or reduce resources during periods of lower demand.
  • High Availability: Kubernetes ensures high availability by distributing containers across multiple nodes in a cluster. Kubernetes can reschedule containers to healthy nodes if a node or container fails to maintain application availability.
  • DevOps and CI/CD Integrations: Kubernetes fits seamlessly into DevOps and CI/CD pipelines, allowing for automated testing, deployment, and scaling of applications.

Users and Uses Cases

  • Enterprises.
  • Startups.
  • DevOps Teams.
  • System Administrator.
  • Internet of Things.

About iptables

Iptables and  ip6tables  are used to set up, maintain, and inspect the tables of IPv4 and IPv6 packet filter rules in the Linux kernel. These tables contain sets of rules, called chains, that will filter incoming and outgoing data packets. These chains are stored in tables. 

Tables and Chains

  • Filter: Used for packet filtering and deciding whether to allow or block packets.

    • INPUT Chain:  for altering packets destined for local sockets on the same device or the server
    • OUTPUT Chain: It deals with outgoing packets generated by processes running on the firewall itself.
    • Forward Chain: handles packets that are routed through the firewall, neither destined for nor generated by the firewall itself.

  • NAT: Used for Network Address Translation, such as port forwarding and address translation.

    • PREROUTING Chain: This chain is used for altering packets as they arrive at the firewall, typically before routing decisions are made. It's often used for destination NAT (DNAT) or port forwarding.
    • POSTROUTING Chain: It processes packets after routing decisions have been made but before they leave the firewall.
    • OUTPUT Chain:  Similar to the Filter Table, but used for NAT-related decisions

  • Mangle: Used for advanced packet-level manipulation and QoS (Quality of Service) settings.

    • PREROUTING Chain: Similar to the NAT Table. But allows more advanced packet manipulation
    • INPUT Chain: Similar to the Filter Table, allowing for more detailed packet alterations.
    • FORWARD Chain: Similar to the Filter Table, is used for packets that are routed through the firewall.
    • OUTPUT Chain:  Similar to the Filter Table, for processing outgoing packets.

Analysis of Kubernetes iptables rules

Closer look

Now let's take a closer look at the chains and rules used for Kubernetes. For this procedure we are using iptables as mentioned above this tool will be useful for this.

iptables -L -t nat | grep -i karmaweb:http
  • -L, to list the rules of the specified tables.
  • -t nat, to specify the table from which the rules are to be displayed.  From the PREROUTING and OUTPUT chains we can see that all the packets incoming and outgoing enter KUBE-SERVICES, set of automatically generated iptables rules that are used to handle service traffic within the cluster.  Let's first look at the KUBE-SERVICES chain which as it is the entry point for service packets, matches the destination IP: port and dispatches the packet to the corresponding KUBE-SVC-* chain.
sudo iptables -L KUBE-SERVICES -t nat | grep karmaweb:http

Since KUBE-SVC-5XYKYCRLOJZB2ITL is the next chain mentioned, so we will inspect that.

sudo iptables -L KUBE-SVC-5XYKYCRLOJZB2ITL -t nat

In KUBE-SVC-5XYKYCRLOJZB2ITL . The rules displayed have 4 parts target, prot, opt, source and destination. We have 2 rules here.

  • First rule 

    • target : It specifies the next checkpoint of the packet which is KUBE-MARK-MASQ, if it satisfies the rules mentioned here.
    • port : This part checks the protocol used by packet. Here the protocol mentioned is tcp.
    • opt: It's the option or flags, it provides how the rule is to be applied for the packet. Here no options are mentioned.
    • source : This will check the source of the packet, the packets not coming from ip-10-224-0-0.ap-south-1.compute.internal/16 domain will be accepted, this domain translates to the IP range 10.244.0.0 to 10.244.255.255. 
    • destination: Specify the destination of the packet., the domain is  ip-10-110-199-117.ap-south-1.compute.internal translates to IP 10.110.199.177 If the packets pass the above rule, it will undergo an alteration process where the source address of the packet will be changed. This takes place in KUBE-POSTROUTING chain.
sudo iptables -L KUBE-POSTROUTING -t nat

  • Second Rule

    • target: The target mentioned here is KUBE-SEP-RX5EM3L6A5YPWQT5, which is another chain.
    • prot: No protocol is specified here. 
    • opt: It is empty as above rulesource: No source is specified.
    • destination : No destination is specified.

KUBE-SVC-* acts as a load balancer and distributes packets to KUBE-SEP-* chains. The number of KUBE-SEP-* chains is equal to the number of endpoints behind the service (i.e. number of running pods). For now, we only have one endpoint. Now the request will be dispatched to KUBE-SEP-RX5EM3L6A5YPWQT5 Next we have to inspect KUBE-SEP-RX5EM3L6A5YPWQT5 chain.

sudo iptables -L KUBE-SEP-RX5EM3L6A5YPWQT5 -t nat

Here we have two rules

  • First rule

    • target : It specifies the next checkpoint of the packet which is KUBE-MARK-MASQ.
    • port : This part checks the protocol used by packet. Here the protocol mentioned is tcp.
    • opt: It's the option or flags, it provides how the rule is to be applied for the packet. Here no options are mentioned.
    • source : No source is specified, It can be from anywhere.
    • destination: Specify the destination of the packet, the domain is  ip-10-244-5-241.ap-south-1.compute.internal translates to IP 10.244.5.241.
    • /* default/karmaweb:http */ : This is a comment that says this rule is related to karamweb service.

  • Second rule 

    • This redirects all the packets to the podIP which is 10.244.5.241:3000 This rule only care about protocol which has to be tcp, source and destination can be anything.

Port Forwarding

The above scenario is an example of port forwarding in Kubernetes. Port forwarding is a concept in networking that enables traffic from the internet to reach a device within a private network. It can be done by router or firewall, so the service hosted in local devices can be accessed from the web.

Experiment on port forwarding

Now we will try to recreate the above scenario using iptables to demonstrate port forwarding. Our goal is to redirect the request done on localhost:8909 to 15.0.0.1:8764. To demonstrate that we are setting up a Python script that will listen on 15.0.0.1:8764. This Python script will be running in a docker container.  

import socket

def fake_service(ip, port):
    server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_socket.bind((ip, port))
    server_socket.listen(1)
    print(f'Fake service is listening on port {port}')
    while True:
        client_socket, client_addr = server_socket.accept()
        print(f'Connection from: {client_addr}')
        http_response = b'HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\nFake service response\n'
        client_socket.send(http_response)
        client_socket.close()

if __name__ == '__main__':
    ip = "localhost"
    port_number = 8765 
    fake_service(ip,port_number)

Find source code here Doing

curl http://localhost:8764

by accessing the terminal of the container . Now we have connect the network of this docker container to the host’s network. By doing this when we do 

curl http://localhost:8764


netstat -tuln

command is used to display active network connections and listening ports on a Linux system. Now we have to create a dummy network interface and assign it an IP, That IP can be used to run the docker container in the HOST.

Creating a Dummy Network Interface

Relation between the network interface and IP In order to connect to the network, a computer must have at least one network interface, Each network interface must have its own unique IP address. The IP address that you give to a host is assigned to its network interface.  If you add a second network interface to a machine, it must have its own unique IP number. Adding a second network interface changes the function of a machine from a host to a router. With the driver now loaded we can create a dummy network interface. A dummy eth10 is created in the host. An IP range is assigned to eth0 also added the label eth10:0 Changing the ip and port variable in the script to 15.0.0.1 and 8764 respectively. 

if __name__ == '__main__':
    ip = "15.0.0.1"
    port_number = 8764 
    fake_service(ip,port_number)

curl gives

curl http://15.0.0.1:8764

When netstat is used the internal  IP and port are displayed, its runs on a different network interface. Now we have to add the required NAT rule to replicate the port forwarding to this IP and port. 

Setting Up iptables

Added an OUTPUT Chain in NAT tables in iptables.

sudo iptables -t nat -A OUTPUT -p tcp --dport 8909 -j DNAT --to-destination 15.0.0.1:8765
  • -t nat: specify the table where the rule has to be added.
  • -A OUTPUT: to append a rule to OUTPUT chain.
  • -p tcp: specify the protocol has to be tcp.
  • --dport 8909: to set the destination port.
  • -j DNAT --to-destination 15.0.0.1:8765: This is the action part of the rule.  This says the packet’s destination IP address and port have to be rewritten to 15.0.0.1:8765 if it meets the above conditions. 

Successful port forwarding

Now when we do 

curl http://localhost:8909

will give the same results as 

curl http://15.0.0.1:8765

This confirms port forwarding is working successfully!

Conclusion

In this article, we explored the complexities of Kubernetes network management, iptables, and port forwarding. We investigate Hexmos infrastructure and discover how Kubernetes hides service ports from traditional tools like netstat. We delve into the fundamentals of Kubernetes, explore the intricacies of iptables, and dissect Kubernetes iptables rules. We demystify port forwarding and demonstrate its practical application. This comprehensive guide sheds light on Kubernetes networking for network administrators and developers in the constantly evolving world of container orchestration.

Hacker News Post

Portless Ports: Demystifying Kubernetes Port Forwarding