I’m trying to build DPDK on Raspberry Pi 3 but couldn’t do it on Raspbian since it’s a 32-bit OS. DPDK uses several assembler instructions for ARM 8 that are only valid on 64-bit OS. So my effort starts with installing a 64-bit OS.
I found SUSE has released 64-bit Linux for Raspberry Pi 3:
I started with this OS image.
Choosing the Image
Three versions of distributions are avalable for Raspberry Pi 3:
- openSUSE Leap
- openSUSE Tumbleweed
- non-upstream openSUSE Tumbleweed
I tried openSUSE Leap first, but could not zypper update. Then I tried openSUSE Tumbleweed. This worked fine.
There are four variations of images:
- JeOS image (Just Enough OS)
- E20 image
- XFCE image
- LXQT image
- X11 image
I wanted to save disk space, so took JeOS.
Installation was done flawlessly. I’m using wired network to avoid spending time for WiFi setup.
The system is missing compiler and so on. I first needed to install development tools. After logging in as root, I executed following.
# zypper refresh # zypper update # zypper install -t pattern devel_C_C++ # zypper install emacs libnuma-devel ncurses-devel kernel-source man
Then the system is ready for building DPDK.
# see http://dpdk.org/dev # and http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html $ git clone git://dpdk.org/dpdk $ cd dpdk $ make install T=arm64-armv8a-linuxapp-gcc
Then compiler has passed, but…
localhost:/home/naoki/workspace/dpdk/arm64-armv8a-linuxapp-gcc/app # ./testpmd ERROR: This system does not support "AES". Please check that RTE_MACHINE is set correctly. EAL: FATAL: unsupported cpu type. EAL: unsupported cpu type. PANIC in main(): Cannot init EAL 1: [./testpmd(rte_dump_stack+0x1c) [0x4c1854]] Aborted (core dumped)
The CPU does not seem to support AES (Advanced Encryption Standard) instructions:
localhost:/home/naoki/workspace/dpdk/arm64-armv8a-linuxapp-gcc/app # lscpu Architecture: aarch64 Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Model: 4 BogoMIPS: 38.40 NUMA node0 CPU(s): 0-3 Flags: fp asimd evtstrm crc32 cpuid
The CPU of Raspberry Pi 3 is Cortex-A53. The Wikipedia page for Arm architecture sounds like Cortex-A53 supports AES instructions, but according to Arm documentation, it seems to be up to implementation. So, I think AES instructions are unsupported by Raspberry Pi 3. How does DPDK think it’s supported then?
I found following code snippet in part of makefiles:
AUTO_CPUFLAGS := $(shell $(CC) $(MACHINE_CFLAGS) $(WERROR_FLAGS) $(EXTRA_CFLAGS) -dM -E - < /dev/null) ... ifneq ($(filter $(AUTO_CPUFLAGS),__ARM_FEATURE_CRYPTO),) CPUFLAGS += AES CPUFLAGS += PMULL CPUFLAGS += SHA1 CPUFLAGS += SHA2 endif
So, gcc returns flag __ARM_FEATURE_CRYPTO that indicates encryption instructions support. But as seen in the previous lscpu output, the CPU does not support it. I guess the compiler gives wrong information because it’s a binary install via zypper. The compiler probably would return correct flags if built from source code, but it’s time consuming. Instead of building gcc, I put a quick workaround to DPDK as follows and rebuilt DPDK.
diff --git a/mk/machine/armv8a/rte.vars.mk b/mk/machine/armv8a/rte.vars.mk index 500421a20..c83045b84 100644 --- a/mk/machine/armv8a/rte.vars.mk +++ b/mk/machine/armv8a/rte.vars.mk @@ -55,4 +55,4 @@ # CPU_LDFLAGS = # CPU_ASFLAGS = -MACHINE_CFLAGS += -march=armv8-a+crc+crypto +MACHINE_CFLAGS += -march=armv8-a+crc diff --git a/mk/rte.cpuflags.mk b/mk/rte.cpuflags.mk index a813c91f4..cb899e86c 100644 --- a/mk/rte.cpuflags.mk +++ b/mk/rte.cpuflags.mk @@ -125,12 +125,12 @@ ifneq ($(filter $(AUTO_CPUFLAGS),__ARM_FEATURE_CRC32),) CPUFLAGS += CRC32 endif -ifneq ($(filter $(AUTO_CPUFLAGS),__ARM_FEATURE_CRYPTO),) -CPUFLAGS += AES -CPUFLAGS += PMULL -CPUFLAGS += SHA1 -CPUFLAGS += SHA2 -endif +# ifneq ($(filter $(AUTO_CPUFLAGS),__ARM_FEATURE_CRYPTO),) +# CPUFLAGS += AES +# CPUFLAGS += PMULL +# CPUFLAGS += SHA1 +# CPUFLAGS += SHA2 +# endif MACHINE_CFLAGS += $(addprefix -DRTE_MACHINE_CPUFLAG_,$(CPUFLAGS))
In this time, execution looks better, although it failed again.
localhost:/home/naoki/workspace/dpdk/arm64-armv8a-linuxapp-gcc/app # ./testpmd EAL: Detected 4 lcore(s) EAL: Probing VFIO support... EAL: No probed ethernet devices USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0 EAL: Error - exiting with code: 1 Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory
I think this is because I haven’t setup huge TLB fs yet.
(To be continued)
PS: This may be helpful later: