MacBook with ARM chip and embedded-libraries problems

Hello everyone, my name is Ihor Sila and I'm a Senior Software Engineer in AUTO1, with over 6 years of developing experience. When I joined AUTO1, I received my laptop and I found out that it's a new MacBook Pro with M1 processor and I believe that's really great, because I've read a lot of articles about this chip's performance and comparisons with Intel i-core series. However when I've tried to build microservices that my team presented to me, I found some unusual situations with tests, which are working with embedded libraries such as redis and postgres.

Embedded-postgres problems

Problem

When I've run first time integration-tests that using embedded-postgres library, I've got next exception in stack trace:

 Factory method 'postgres' threw  exception; nested exception is
 java.lang.IllegalStateException: Process
 [/var/folders/61/ls8bv20x4w3fsn3wpm8dlpfm0000gp/T/embedded-pg/
 PG-0c0585587b40695247cf72650cd42f92/bin/initdb

Solution

With the postgres problem, the solution was pretty simple and after I found it in our Slack channel I applied it and it solved the problem. So, when you are setting up your Mac, please check the environment variables and switch locale to en_US.UTF-8 from default one, because the embedded postgres library tries to start its library with a different locale and has problems with the process id. Another solution is to increase the size of a single shared memory segment a Linux process can allocate. So the commands in terminal will be the next

sudo sysctl kern.sysv.shmmax=104857600
sudo sysctl kern.sysv.shmall=25600
sudo sysctl kern.sysv.shmmin=1
sudo sysctl kern.sysv.shmmni=10240
sudo sysctl kern.sysv.shmseg=4096

export LC_NUMERIC=en_US.UTF-8
export LC_MONETARY=en_US.UTF-8
export LC_TIME=en_US.UTF-8
export LC_ALL=en_US.UTF-8
export LC_COLLATE=en_US.UTF-8
export LC_MESSAGES=en_US.UTF-8
export LC_CTYPE=en_US.UTF-8
export LANG=en_US.UTF-8

And that you should restart your MacBook to apply all settings. After that the problem should gone.

Embedded-redis problems

Problem

Moving on to redis, the problem was that it was newly added to AUTO1. So, from a code perspective everything is fine, but when you try to run the integration tests that are using embedded redis, they just won't be executed, because redis doesn't want to start. There is some exception log that you receive but that's about it, nothing special.

java.lang.RuntimeException: Can't start redis server. Check
logs for details.

at
redis.embedded.AbstractRedisInstance.awaitRedisServerReady
(AbstractRedisInstance.java:61)
at
redis.embedded.AbstractRedisInstance.start
(AbstractRedisInstance.java:39)
at redis.embedded.RedisServer.start(RedisServer.java:9)
at
wkda.common.distributed.cache.redis.EmbeddedSingleRedisServer
.before(EmbeddedSingleRedisServer.java:48)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner
.run(SpringJUnit4ClassRunner.java:190)

Deep diving into problem

When I've tried to debug the library code, it also appeared fine from a first look, but when I've debugged the stack trace of the startup, I've realized that this old version of redis doesn't know anything about the new M1 architecture and that macOS can be built on something different other than an Intel chip.

=== REDIS BUG REPORT START: Cut & paste starting  from here ===
30203:M 15 Feb 14:01:14.039 # Redis 3.2.4 crashed by signal: 10
30203:M 15 Feb 14:01:14.040 # Crashed running the instuction
at: 0x7fff20406440
30203:M 15 Feb 14:01:14.040 # Accessing address: 0x305685000
30203:M 15 Feb 14:01:14.040 # Failed  assertion: <no
assertion  failed> (<no  file>:0)

------ STACK TRACE ------
EIP:
0 libsystem_platform.dylib 0x00007fff20406440
_platform_memset$VARIANT$Rosetta + 108

Backtrace:
0 redis-server.app 0x0000000100a70a7e logStackTrace + 110
1 redis-server.app 0x0000000100a70e4d sigsegvHandler + 253
2 libsystem_platform.dylib 0x00007fff20403d7d _sigtramp + 29
3 ??? 0x00000001091c1200  0x0 + 4447801856
4 ??? 0x31207070612e7265  0x0 + 3539952935082291813

Basically, old redis library only knows that macOS can be either x86 or x86_64. After a thorough investigation, I realized that there is no solution yet and repositories that provide embedded redis solutions haven't updated their libraries so far.

Solution

The solution to this problem consists of two parts:

1. The first part is to somehow find the redis application that can be started up on both macOS with M1 chip and on Intel chip, gracefully.
2. The second one is to replace the old redis-application with a new one.

To solve the first problem, I thought that I could just download the latest redis-server, then get the application file from the library folder and replace the old app file with the new one. Soon after the first try I've realized that my solution will work only for M1 chips, therefore no compatibility with the Intel chipped Macs.

I've dived deeper into the redis library to understand how it knows which architecture (x86 or x86_64) it needs to use. The solution was pretty simple, it just executes the command to get system flags and then analyzes these flags to see which version to use. When I executed the same command on my local machine, I realized that on M1 architecture some new flags were added, which represents that the current architecture is written with a new chip. In embedded redis you can tell redis to use another embedded redis application, but not the solution from the library.

If you execute next command at the terminal, you will found extra flags for AMD architecture:

sysctl -a | grep hw.optional

------- Result -------

hw.optional.amx_version: 2
hw.optional.arm64: 1
hw.optional.armv8_1_atomics: 1
hw.optional.armv8_2_fhm: 1
hw.optional.armv8_2_sha3: 1
hw.optional.armv8_2_sha512: 1
hw.optional.armv8_crc32: 1
hw.optional.breakpoint: 6
hw.optional.floatingpoint: 1
hw.optional.neon: 1
hw.optional.neon_fp16: 1
hw.optional.neon_hpfp: 1
hw.optional.ucnormal_mem: 1
hw.optional.watchpoint: 4

But if you will execute the same command on Intel-based architecture, you will see another flags, that are specific for Intel architecture:

sysctl -a | grep hw.optional
hw.optional.floatingpoint: 1
hw.optional.mmx: 1
hw.optional.sse: 1
hw.optional.sse2: 1
hw.optional.sse3: 1
hw.optional.supplementalsse3: 1
hw.optional.sse4_1: 1
hw.optional.sse4_2: 1
hw.optional.x86_64: 1
hw.optional.aes: 1
hw.optional.avx1_0: 1
hw.optional.rdrand: 1
hw.optional.f16c: 1
hw.optional.enfstrg: 1
hw.optional.fma: 1
hw.optional.avx2_0: 1
hw.optional.bmi1: 1
hw.optional.bmi2: 1
hw.optional.rtm: 0
hw.optional.hle: 0
hw.optional.adx: 1
hw.optional.mpx: 0
hw.optional.sgx: 0
hw.optional.avx512f: 0
hw.optional.avx512cd: 0
hw.optional.avx512dq: 0
hw.optional.avx512bw: 0
hw.optional.avx512vl: 0
hw.optional.avx512ifma: 0
hw.optional.avx512vbmi: 0

After realizing that, we created a class so that we can tell the library the fact that if the OS is macOS, then it needs to analyze the system flags. If it will find M1-specific flags (which are not present on Intel architecture), then use the latest redis application file which can be executed on macOS with M1(AMD) architecture, otherwise use the existing solution, because it's working on macOS with Intel processor. We also should store this application file in our resource folder, so in that case new developers can just download the source code and the redis application file will be already there.

protected  void before() {
RedisExecProvider provider = RedisExecProvider.defaultProvider()
.override(OS.UNIX, "redis-server/redis-server")
.override(OS.WINDOWS, Architecture.x86,
"redis-server/redis-server.exe")
.override(OS.WINDOWS, Architecture.x86_64,
"redis-server/redis-server.exe")
.override(OS.MAC_OS_X, Architecture.x86,
"redis-server/redis-server.app")
.override(OS.MAC_OS_X, Architecture.x86_64,
getRedisServerForAppleArchitecture());


this.redisServer = RedisServer.builder()
.redisExecProvider(provider)
.port(redisPort)
.setting("bind " + this.redisHost)
.build();

this.redisServer.start();
log.info("Started Redis server at [{}:{}]",
this.redisHost, this.redisPort);
}


private  String getRedisServerForAppleArchitecture()
throws Exception {
Process process = Runtime.getRuntime().exec("sysctl hw");
try (BufferedReader input = new  BufferedReader(new
InputStreamReader(process.getInputStream()))) {
String  line;
while((line = input.readLine()) != null) {
if (line.contains("hw.optional.arm64") &&
line.trim().endsWith("1")) {
return  "redis-server/amd64/redis-server.app";
}
}
} catch (IOException e) {
log.error("Error during parsing system process");
throw  new RuntimeException(e);
}
return  "redis-server/redis-server.app";
}

We've tested this approach on our microservices and it's stable.

As for now, our core-team already replaced dependency from old embedded-redis to new one and it's working for both Intel and AMD architecture without additional changes.

Thanks for reading!