Skip to content

SmartNIC: Install popcorn kernel on the SmartNIC setup

Jump to bottom
Gautam Sharma edited this page Dec 20, 2022 · 3 revisions

Install and run popcorn-linux on the SmartNIC setup

Overview

This guide describes how to set up a testing environment for Popcorn Linux using an x86 host equipped with a SmartNIC. Basically, we will run popcorn on 2 machines - one on x86 host and one for ARM SmartNIC, where the host and the SmartNIC are connected over PCI-e connection. The examples and commands are based on Ubuntu 20.04.

--------------   --------------
|   Node 0   |   |   Node 1   |
|    Host    |   |  SmartNIC  |
|    x86     |   |    arm     |
| 10.4.4.100 |   | 10.4.4.101 |
--------------   --------------
-------------------------------
|  linux-x86      linux-arm   |
|                             |
|           host (x86)        |
|           10.4.4.10         |
-------------------------------

Setup the Host network interface

  • Backup default netplan config
# cp /etc/netplan/01-netcfg.yaml ~/default_nw_cfg.yaml
  • Install bridge-utils package
# apt-get install bridge-utils
  • Write the network configuration Replace enp1s0f0 with your main network interface
# vi /etc/netplan/01-netcfg.yaml
network:
    version: 2
    renderer: networkd
    ethernets:
        enp1s0f0:
            dhcp4: yes
        tmfifo_net0:
            dhcp4: no
            addresses: [10.4.4.100/24]
    bridges:
        br0:
            dhcp4: no
            addresses: [10.4.4.10/24]
            gateway4: 10.4.4.1
            nameservers:
                addresses:
                - 8.8.8.8
            interfaces:
            - enp1s0f0
  • Apply netplan configuration
# netplan apply
  • Check the network configuration
# ifconfig

The output should reflect the network configuration that we have written.

  • Note: In this example we have assigned IPs for a specific subnet. Please use IPs that belong in the same subnet as your network.

Setup the SmartNIC network interface

  • Disable cloud-init network configuration

Write network: {config: disabled} to the 99-disable-network-config.cfg file.

# vi /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
network: {config: disabled}
  • Write the network configuration for 00-nm.yaml 
# vi /etc/netplan/00-nm.yaml
network:
    version: 2
    renderer: NetworkManager
  • Write the network configuration for 50-cloud-init.yaml 
# vi /etc/netplan/50-clout-init.yaml
network:
    ethernets:
        tmfifo_net0:
            addresses:
            - 10.4.4.101/24
            dhcp4: false
            nameservers:
                addresses:
                - 10.4.4.100
            routes:
            - metric: 1025
              to: 0.0.0.0/0
              via: 10.4.4.100
    renderer: networkd
    version: 2
  • Apply netplan configuration
# netplan apply
  • Check the network configuration
# ifconfig

The output should reflect the network configuration that we have written.

  • Note: In this example we have assigned IPs for a specific subnet. Please use IPs that belong in the same subnet as your network.

Prepare for the setup on Host

  • Let's start with updating apt repository and install some utilities:
$ sudo apt-get update
$ sudo apt-get install build-essential libssl-dev libncursesw5-dev git curl bc
  • Retrieve the popcorn kernel source code
$ git clone --depth=1 -b main --single-branch https://github.com/ssrg-vt/popcorn-kernel.git
$ cd popcorn-kernel
  • Kernel config
$ cat /boot/config-$(uname -r) > .config

Note: You can use your own config but make sure the following options are disabled: - CONFIG_SWAP - CONFIG_TRANSPARENT_HUGEPAGE - CONFIG_CMA - CONFIG_MIGRATION - CONFIG_COMPACTION - CONFIG_KSM - CONFIG_MEM_SOFT_DIRTY Page should be 4 KB. ARM should select ARM64_4K_PAGES PPC should use CONFIG_PPC_4K_PAGES.

  • Add popcorn-specific config options to the end of your .config file
#
# Popcorn Distributed Execution Support
#
CONFIG_ARCH_SUPPORTS_POPCORN=y
CONFIG_POPCORN=y
CONFIG_POPCORN_DEBUG=y
CONFIG_POPCORN_DEBUG_PROCESS_SERVER=y
# CONFIG_POPCORN_DEBUG_PAGE_SERVER is not set
# CONFIG_POPCORN_DEBUG_VMA_SERVER is not set
# CONFIG_POPCORN_DEBUG_VERBOSE is not set
CONFIG_POPCORN_CHECK_SANITY=y
# CONFIG_POPCORN_REMOTE_INFO is not set
CONFIG_POPCORN_STAT=y
# CONFIG_POPCORN_STAT_PGFAULTS is not set
CONFIG_POPCORN_KMSG=y
CONFIG_POPCORN_KMSG_SOCKET=m
CONFIG_POPCORN_KMSG_RDMA=m
CONFIG_POPCORN_KMSG_TEST=m
CONFIG_POPCORN_DEBUG_MSG_LAYER=y
  • Update the .config file
$./update_config.sh
  • Build the kernel
$ make -j 8
$ sudo make modules_install
$ sudo make install
  • Reboot with popcorn kernel and check using
uname -r

Access the SmartNIC via Host

  • From now on, we will run the commonds on the host.

Apply the network configuration

# netplan apply
  • If there is an error, due to typos or whatever, use the command in debug mode to find out more details:
# netplan --debug apply

Enable IP forwarding on the host

# echo 1 | tee /proc/sys/net/ipv4/ip_forward

Enable NAT on the internet-facing interface on the host

# iptables -t nat -A POSTROUTING -o enp1s0f0 -j MASQUERADE
  • Note: Replace enp1s0f0 with your main (internet-facing) interface.

IP forwarding

# net.ipv4.conf.eno1.forwarding = 1
  • If this is indeed enabled, but still no Internet access, then it might have happened that your default firewall rules are configured to an implicit deny policy. Let’s change that:
# iptables -A FORWARD -o enp1s0f0 -j ACCEPT
# iptables -A FORWARD -m state --state ESTABLISHED,RELATED -i enp1s0f0 -j ACCEPT

Set Google's DNS on the BlueField

# echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf

Full internet access is enabled

# ping google.com

Test the network

  • Try pinging the SmartNIC
# ping -I tmfifo_net0 192.168.100.2 -c2
  • You should be able to ping the SmartNIC at this point.

Connect to the SmartNIC via Host

# ssh <username>@10.4.4.101

Prepare for the setup on SmartNIC

  • Make sure you are ssh-ed into the SmartNIC before you proceed.
  • Let's start with updating apt repository and install some utilities:
$ sudo apt-get update
$ sudo apt-get install build-essential libssl-dev libncursesw5-dev git curl bc
  • Retrieve the popcorn kernel source code
$ git clone --depth=1 -b main --single-branch https://github.com/ssrg-vt/popcorn-kernel.git
$ cd popcorn-kernel
  • Kernel config
$ cat /boot/config-$(uname -r) > .config

Note: You can use your own config but make sure the following options are disabled: - CONFIG_SWAP - CONFIG_TRANSPARENT_HUGEPAGE - CONFIG_CMA - CONFIG_MIGRATION - CONFIG_COMPACTION - CONFIG_KSM - CONFIG_MEM_SOFT_DIRTY Page should be 4 KB. ARM should select ARM64_4K_PAGES PPC should use CONFIG_PPC_4K_PAGES.

  • Add popcorn-specific config options to the end of your .config file
#
# Popcorn Distributed Execution Support
#
CONFIG_ARCH_SUPPORTS_POPCORN=y
CONFIG_POPCORN=y
CONFIG_POPCORN_DEBUG=y
CONFIG_POPCORN_DEBUG_PROCESS_SERVER=y
# CONFIG_POPCORN_DEBUG_PAGE_SERVER is not set
# CONFIG_POPCORN_DEBUG_VMA_SERVER is not set
# CONFIG_POPCORN_DEBUG_VERBOSE is not set
CONFIG_POPCORN_CHECK_SANITY=y
# CONFIG_POPCORN_REMOTE_INFO is not set
CONFIG_POPCORN_STAT=y
# CONFIG_POPCORN_STAT_PGFAULTS is not set
CONFIG_POPCORN_KMSG=y
CONFIG_POPCORN_KMSG_SOCKET=m
CONFIG_POPCORN_KMSG_RDMA=m
CONFIG_POPCORN_KMSG_TEST=m
CONFIG_POPCORN_DEBUG_MSG_LAYER=y
  • Update the .config file
$./update_config.sh
  • Build the kernel
$ make -j 4
$ sudo make modules_install
$ sudo make install
  • Reboot with popcorn kernel and check using
uname -r
  • Note: You may choose to cross-compile the kernel for the SmartNIC on the Host.

Setup the Popcorn messaging layer

  • Note: Before we proceed, it is necessary to run these commands before loading the messaging layer module. Run these commands everytime before loading the messaging layer module.
$ sudo netplan apply
$ sudo echo 1 | tee /proc/sys/net/ipv4/ip_forward
$ sudo iptables -t nat -A POSTROUTING -o enp1s0f0 -j MASQUERADE
$ sudo net.ipv4.conf.eno1.forwarding = 1
$ sudo iptables -A FORWARD -o enp1s0f0 -j ACCEPT
$ sudo iptables -A FORWARD -m state --state ESTABLISHED,RELATED -i enp1s0f0 -j ACCEPT

Turn the above mentioned set of commands into a script so that you can run it everytime before you load messaging layer module.

  • Messaging layer module Popcorn Linux is comprised of many subsystems and a messaging module. The messaging module contains the network and system configurations (in this example, a machine at 10.4.4.100 is node 0, a machine at 10.4.4.101 is node 1), and the rest of Popcorn subsystems are initialized only after the messaging module is successfully loaded. When we insert the messaging module into the kernel, Popcorn identifies itself (e.g., I am node 0 on x86 as my IP address is 10.4.4.100) and establishes connections to all nodes in the cluster (node 1 in this case). Currently such configurations are hardcoded in the ip_addresses[] array in msg_layer/config.h in the source code in Popcorn4.4. Pocorn5.2's configuration is in /etc/popcorn/nodes. All kernels in the same Popcorn "cluster" should have the identical configuration.

  • (Popcorn Linux4.4 only) Let check the configuration from popcorn-kernel/msg_layer/config.h. Edit the IP addresses accordingly if you are using your IP address set:

const char *ip_addresses[] = {
    "10.4.4.100", /* x86 address */
    "10.4.4.101", /* arm address */
}
        ...

Note: Make the same changes to the config.h file on the SmartNIC as well.

  • (Popcorn Linux5.2 only) Create a config file called "nodes" under "/etc/popcorn" on x86 server.
popcorn@x86:~$ cat /etc/popcorn/nodes
10.4.4.100
10.4.4.101
  • Compile the message layer module for x86:
$ make -j 8 msg_layer/msg_socket.ko
        ...
  LD [M]  msg_layer/msg_socket.ko
  • Copy to module to home directory
$ cp msg_layer/msg_socket.ko ~
  • Compile the message layer module for SmartNIC:
$ ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" make -j 8 msg_layer/msg_socket.ko
        ...
  LD [M]  msg_layer/msg_socket.ko
  • Copy to module to SmartNIC's home directory
$ scp msg_layer/msg_socket.ko <username>@10.4.4.101:
  • Load the module on Host (x86)
$ sudo insmod ./msg_socket.ko
  • In a new terminal instance, load the module on SmartNIC (arm)
$ ssh <username>@10.4.4.101
$ sudo insmod ./msg_socket.ko

Congratulations! You have successfully installed the popcorn kernel on the host as well as the SmartNIC and tested the setup!

Run NPB Benchmarks

  • Follow the instructions to run NPB benchmarks using the Compiler Setup guide.

Notes

  • Make sure the firewall program ufw is not blocking tmfifo_net0 or any other IPs
  • To disable ufw, use sudo ufw disable. Disabling ufw is not recommended but you can try that for testing purposes.
  • If there is lack of space on the SmartNIC, use other partitions on it. There are 3-4 paritions on the SmartNIC that are always empty. Alternatively, set the partitions manually.
Clone this wiki locally