musl C++ support

[anguslees]

Trying to build a Rust project that uses C++ (via gcc-rs) on one of the musl targets fails with "musl-g++ not found". It would be neat if this combination just worked as well as C++ on other targets.

[Timmmm]

We also have this problem (well, it says musl-gcc not found), but only on Windows, and only recently. Haven't narrowed down the cause yet unfortunately.

[brainstorm]

Would you mind trying this newer version of MUSL and let me know if it works for you now?: #452

[Hoverbear]

@brainstorm It seems that the x86_64 image can now build C++ code, but it doesn't link to the libstdc++. :(

Here's an error sample without changing this main branch:

# ..
          /cargo/git/checkouts/leveldb-sys-23b2a41b0af71d11/bbdd99a/deps/leveldb-1.22/table/format.cc:32: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::size() const'
# ...
          /usr/local/x86_64-linux-musl/include/c++/9.2.0/bits/vector.tcc:643: undefined reference to `__cxa_begin_catch'
          /usr/local/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/local/x86_64-linux-musl/include/c++/9.2.0/bits/vector.tcc:646: undefined reference to `__cxa_rethrow'
          /usr/local/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/local/x86_64-linux-musl/include/c++/9.2.0/bits/vector.tcc:643: undefined reference to `__cxa_end_catch'
          /usr/local/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/local/x86_64-linux-musl/include/c++/9.2.0/bits/vector.tcc:663: undefined reference to `__cxa_begin_catch'
          /usr/local/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/local/x86_64-linux-musl/include/c++/9.2.0/bits/vector.tcc:669: undefined reference to `__cxa_rethrow'
          /usr/local/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/local/x86_64-linux-musl/include/c++/9.2.0/bits/vector.tcc:663: undefined reference to `__cxa_end_catch'
          /usr/local/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /target/x86_64-unknown-linux-musl/debug/deps/libleveldb_sys-f89a547530e5179d.rlib(filter_block.cc.o): in function `void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&)':
          /usr/local/x86_64-linux-musl/include/c++/9.2.0/bits/vector.tcc:485: undefined reference to `__cxa_begin_catch'
# ...
          /usr/local/x86_64-linux-musl/include/c++/9.2.0/ext/new_allocator.h:128: undefined reference to `operator delete(void*)'
          /usr/local/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /target/x86_64-unknown-linux-musl/debug/deps/libleveldb_sys-f89a547530e5179d.rlib(env_posix.cc.o): in function `__gnu_cxx::new_allocator<leveldb::(anonymous namespace)::PosixEnv::BackgroundWorkItem>::allocate(unsigned long, void const*)':
          /usr/local/x86_64-linux-musl/include/c++/9.2.0/ext/new_allocator.h:105: undefined reference to `std::__throw_bad_alloc()'
          /usr/local/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/local/x86_64-linux-musl/include/c++/9.2.0/ext/new_allocator.h:114: undefined reference to `operator new(unsigned long)'
          /usr/local/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /target/x86_64-unknown-linux-musl/debug/deps/libleveldb_sys-f89a547530e5179d.rlib(env_posix.cc.o): in function `__gnu_cxx::new_allocator<leveldb::(anonymous namespace)::PosixEnv::BackgroundWorkItem*>::allocate(unsigned long, void const*)':
          /usr/local/x86_64-linux-musl/include/c++/9.2.0/ext/new_allocator.h:105: undefined reference to `std::__throw_bad_alloc()'
          /usr/local/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /usr/local/x86_64-linux-musl/include/c++/9.2.0/ext/new_allocator.h:114: undefined reference to `operator new(unsigned long)'
          /usr/local/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /target/x86_64-unknown-linux-musl/debug/deps/libleveldb_sys-f89a547530e5179d.rlib(env_posix.cc.o): in function `__gnu_cxx::new_allocator<std::_Rb_tree_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::allocate(unsigned long, void const*)':

And if we make a small tweak:

  FROM rustembedded/cross:x86_64-unknown-linux-musl

+ ENV CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_RUSTFLAGS="-C link-arg=-lstdc++"

Then we see this, the build working, but dynamic links being produced:

ana@autonoma:~/git/timberio/vector$ lddtree target/x86_64-unknown-linux-musl/debug/vector
vector => target/x86_64-unknown-linux-musl/debug/vector (interpreter => none)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
        ld-linux-x86-64.so.2 => /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1

We're looking to adopt Cross on https://vector.dev/ and this is a hangup for us.

[brainstorm]

@Hoverbear Have you tried the passthrough= Cross.toml directive? That could avoid building your own docker container, i.e:

https://github.com/brainstorm/s3-rust-htslib-bam/blob/master/Cross.toml

You can also try to pass -static to your linker if you don't want dynamic linking, i.e:

https://github.com/alexcrichton/xz2-rs/pull/63/files

[Hoverbear]

@brainstorm I saw that, too! We are using that for some variable things. :) I will probably do that if I end up finding a solution to this linking issue! Thanks for letting me know about it.

I was trying the -static and -static-lib and such a bit today, for example:

But these didn't result in success. As far as I can tell, using a musl target implies -static generally? I also tried injecting things like link-lib without much success.

I suspect in our case the problem lies in https://github.com/timberio/leveldb-sys/blob/bbdd99a6fab0a7444a8c6eb0cc86a1a5a75f7fa1/src/build.rs#L35-L53 , however I've also tried the upstream which does not use cmake and it's been encountering similar issues:

https://github.com/skade/leveldb-sys/blob/b4944d90d3f83bc43e0ce558a776824d0b44e909/src/build.rs#L35-L52

[brainstorm]

Yeah, in order for the whole build to be -static, all the deps and *-sys deps (including transitive ones) must be compiled statically (w/ musl) too, AFAIK :-S

Keep me posted, this is interesting, I wrestled with C -static w/ MUSL but not with C++ yet :)

[Hoverbear]

Yup, we've been wrestling with it too!

So far we've isolated it to leveldb-sys, so I'm going to be focusing on that and trying to figure it out. I'll keep you updated!

We use https://github.com/timberio/vector/blob/master/scripts/ci-docker-images/builder-x86_64-unknown-linux-musl/Dockerfile, but we would prefer to use cross, of course. :)

If you want to chat interactively you can catch me on our vector community space https://discord.gg/phwDQpd or on Twitter (we're mutuals now!).

[Alexhuszagh]

Closing since this is complete for all targets but x86_64-unknown-linux-musl as of #905 and this remaining target is a duplicate of #902, and #817 will completely fix this.