Building DPDK on Raspberry Pi 3

I’m trying to build DPDK on Raspberry Pi 3 but couldn’t do it on Raspbian since it’s a 32-bit OS. DPDK uses several assembler instructions for ARM 8 that are only valid on 64-bit OS. So my effort starts with installing a 64-bit OS.

I found SUSE has released 64-bit Linux for Raspberry Pi 3:
https://en.opensuse.org/HCL:Raspberry_Pi3

I started with this OS image.

Choosing the Image

Three versions of distributions are avalable for Raspberry Pi 3:

• openSUSE Leap
• openSUSE Tumbleweed
• non-upstream openSUSE Tumbleweed

I tried openSUSE Leap first, but could not zypper update. Then I tried openSUSE Tumbleweed. This worked fine.

There are four variations of images:

• JeOS image (Just Enough OS)
• E20 image
• XFCE image
• LXQT image
• X11 image

I wanted to save disk space, so took JeOS.

Installation was done flawlessly. I’m using wired network to avoid spending time for WiFi setup.

Build Preparation

The system is missing compiler and so on. I first needed to install development tools. After logging in as root, I executed following.

# zypper refresh
# zypper update
# zypper install -t pattern devel_C_C++
# zypper install emacs libnuma-devel ncurses-devel kernel-source man

Then the system is ready for building DPDK.

Building DPDK

# see http://dpdk.org/dev
# and http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html
$ git clone git://dpdk.org/dpdk
$ cd dpdk
$ make install T=arm64-armv8a-linuxapp-gcc

Then compiler has passed, but…

Test Run

localhost:/home/naoki/workspace/dpdk/arm64-armv8a-linuxapp-gcc/app # ./testpmd 
ERROR: This system does not support "AES".
Please check that RTE_MACHINE is set correctly.
EAL: FATAL: unsupported cpu type.
EAL: unsupported cpu type.
PANIC in main():
Cannot init EAL
1: [./testpmd(rte_dump_stack+0x1c) [0x4c1854]]
Aborted (core dumped)

The CPU does not seem to support AES (Advanced Encryption Standard) instructions:

localhost:/home/naoki/workspace/dpdk/arm64-armv8a-linuxapp-gcc/app # lscpu
Architecture:        aarch64
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Model:               4
BogoMIPS:            38.40
NUMA node0 CPU(s):   0-3
Flags:               fp asimd evtstrm crc32 cpuid

The CPU of Raspberry Pi 3 is Cortex-A53. The Wikipedia page for Arm architecture sounds like Cortex-A53 supports AES instructions, but according to Arm documentation, it seems to be up to implementation. So, I think AES instructions are unsupported by Raspberry Pi 3. How does DPDK think it’s supported then?

I found following code snippet in part of makefiles:

AUTO_CPUFLAGS := $(shell $(CC) $(MACHINE_CFLAGS) $(WERROR_FLAGS) $(EXTRA_CFLAGS) -dM -E - < /dev/null)

...

ifneq ($(filter $(AUTO_CPUFLAGS),__ARM_FEATURE_CRYPTO),)
CPUFLAGS += AES
CPUFLAGS += PMULL
CPUFLAGS += SHA1
CPUFLAGS += SHA2
endif

So, gcc returns flag __ARM_FEATURE_CRYPTO that indicates encryption instructions support. But as seen in the previous lscpu output, the CPU does not support it. I guess the compiler gives wrong information because it’s a binary install via zypper. The compiler probably would return correct flags if built from source code, but it’s time consuming. Instead of building gcc, I put a quick workaround to DPDK as follows and rebuilt DPDK.

diff --git a/mk/machine/armv8a/rte.vars.mk b/mk/machine/armv8a/rte.vars.mk
index 500421a20..c83045b84 100644
--- a/mk/machine/armv8a/rte.vars.mk
+++ b/mk/machine/armv8a/rte.vars.mk
@@ -55,4 +55,4 @@
 # CPU_LDFLAGS =
 # CPU_ASFLAGS =
 
-MACHINE_CFLAGS += -march=armv8-a+crc+crypto
+MACHINE_CFLAGS += -march=armv8-a+crc
diff --git a/mk/rte.cpuflags.mk b/mk/rte.cpuflags.mk
index a813c91f4..cb899e86c 100644
--- a/mk/rte.cpuflags.mk
+++ b/mk/rte.cpuflags.mk
@@ -125,12 +125,12 @@ ifneq ($(filter $(AUTO_CPUFLAGS),__ARM_FEATURE_CRC32),)
 CPUFLAGS += CRC32
 endif
 
-ifneq ($(filter $(AUTO_CPUFLAGS),__ARM_FEATURE_CRYPTO),)
-CPUFLAGS += AES
-CPUFLAGS += PMULL
-CPUFLAGS += SHA1
-CPUFLAGS += SHA2
-endif
+# ifneq ($(filter $(AUTO_CPUFLAGS),__ARM_FEATURE_CRYPTO),)
+# CPUFLAGS += AES
+# CPUFLAGS += PMULL
+# CPUFLAGS += SHA1
+# CPUFLAGS += SHA2
+# endif
 
 MACHINE_CFLAGS += $(addprefix -DRTE_MACHINE_CPUFLAG_,$(CPUFLAGS))

In this time, execution looks better, although it failed again.

localhost:/home/naoki/workspace/dpdk/arm64-armv8a-linuxapp-gcc/app # ./testpmd 
EAL: Detected 4 lcore(s)
EAL: Probing VFIO support...
EAL: No probed ethernet devices
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0
EAL: Error - exiting with code: 1
  Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory

I think this is because I haven’t setup huge TLB fs yet.

(To be continued)

PS: This may be helpful later:

https://www.suse.com/communities/blog/compiling-de-linux-kernel-suse-way/