RISCV64 on Alpine (not on Ubuntu): ../gf16/gf16_rvv_common.h:33:20: error: incompatible type for argument 3 of '__riscv_vxor_vx_i16m1_m' #62

sanderjo · 2024-08-22T09:21:52Z

Trying to build ParPar in a docker (alpine:edge, gcc 14) on RISCV64, I get the output with errors below

As it's in rvv, I'm assuming it's ParPar specific (not something upstream)?

Is this something in the ParPar source code, or on my side?

  CC(target) Release/obj.target/gf16_rvv/gf16/gf16_shuffle128_rvv.o
In file included from ../gf16/gf16_shuffle128_rvv.c:1:
../gf16/gf16_rvv_common.h: In function 'gf16_vec_mul2_rvv':
../gf16/gf16_rvv_common.h:33:20: error: incompatible type for argument 3 of '__riscv_vxor_vx_i16m1_m'
   33 |                 v, v,
      |                    ^
      |                    |
      |                    vint16m1_t
In file included from ../gf16/gf16_rvv_common.h:9:
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/include/riscv_vector.h:40:9: note: expected 'short int' but argument is of type 'vint16m1_t'
   40 | #pragma riscv intrinsic "vector"
      |         ^~~~~
../gf16/gf16_rvv_common.h:13:17: error: too many arguments to function '__riscv_vxor_vx_i16m1_m'
   13 | #  define RV(f) __riscv_##f
      |                 ^~~~~~~~
../gf16/gf16_rvv_common.h:29:16: note: in expansion of macro 'RV'
   29 |         return RV(vxor_vx_i16m1_m)
      |                ^~
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/include/riscv_vector.h:40:9: note: declared here
   40 | #pragma riscv intrinsic "vector"
      |         ^~~~~
../gf16/gf16_shuffle128_rvv.c: In function '_vlseg2e8':
../gf16/gf16_shuffle128_rvv.c:16:38: error: 'vd' undeclared (first use in this function); did you mean 'd'?
   16 |         *v0 = RV(vget_v_u8m1x2_u8m1)(vd, 0);
      |                                      ^~
      |                                      d
../gf16/gf16_shuffle128_rvv.c:16:38: note: each undeclared identifier is reported only once for each function it appears in
../gf16/gf16_shuffle128_rvv.c:15:22: warning: unused variable 'd' [-Wunused-variable]
   15 |         vuint8m1x2_t d = RV(vlseg2e8_v_u8m1x2)(src, vl);
      |                      ^
In file included from ../gf16/gf16_shuffle128_rvv.c:244:
../gf16/gf16_checksum_rvv.h: In function 'gf16_checksum_exp_rvv':
../gf16/gf16_checksum_rvv.h:71:35: error: incompatible type for argument 4 of '__riscv_vxor_vv_i16m1_m'
   71 |                         res, res, _checksum,
      |                                   ^~~~~~~~~
      |                                   |
      |                                   vint16m1_t
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/include/riscv_vector.h:40:9: note: expected 'long unsigned int' but argument is of type 'vint16m1_t'
   40 | #pragma riscv intrinsic "vector"
      |         ^~~~~
../gf16/gf16_rvv_common.h:13:17: error: too many arguments to function '__riscv_vxor_vv_i16m1_m'
   13 | #  define RV(f) __riscv_##f
      |                 ^~~~~~~~
../gf16/gf16_checksum_rvv.h:67:23: note: in expansion of macro 'RV'
   67 |                 res = RV(vxor_vv_i16m1_m)
      |                       ^~
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/include/riscv_vector.h:40:9: note: declared here
   40 | #pragma riscv intrinsic "vector"
      |         ^~~~~
../gf16/gf16_rvv_common.h: In function 'gf16_vec_mul2_rvv':
../gf16/gf16_rvv_common.h:37:1: warning: control reaches end of non-void function [-Wreturn-type]
   37 | }
      | ^
make: *** [gf16_rvv.target.mk:121: Release/obj.target/gf16_rvv/gf16/gf16_shuffle128_rvv.o] Error 1
make: Leaving directory '/ParPar/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.<anonymous> (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:209:23)
gyp ERR! System Linux 6.1.15
gyp ERR! command "/usr/bin/node" "/usr/bin/node-gyp" "rebuild"
gyp ERR! cwd /ParPar
gyp ERR! node -v v20.15.1
gyp ERR! node-gyp -v v10.1.0
gyp ERR! not ok 
The command '/bin/sh -c git clone https://github.com/animetosho/ParPar.git && 	cd ParPar && 	node-gyp rebuild && 	npm install' returned a non-zero code: 1

The text was updated successfully, but these errors were encountered:

sanderjo · 2024-08-22T09:24:15Z

Dockerfile:

FROM alpine:edge


RUN apk update
RUN apk add nodejs npm python3 gcc g++ make automake autoconf git
RUN \
	git clone https://github.com/animetosho/ParPar.git && \
	cd ParPar && \
	node-gyp rebuild && \
	npm install

Full docker build output attached

ParPar-docker-build.txt

sanderjo · 2024-08-22T09:58:25Z

FWIW: on x86, the same Dockerfile results with success, with parpar working

/ParPar/bin # fallocate -l100M blabla.bin

/ParPar/bin # node parpar.js  -s2000 --min-input-slices=1 -r10% -d pow2 -o parfiles *bin
Input data        : 100 MiB (2000 slices from 1 file)
Recovery data     : 10 MiB (200 * 51.2 KiB slices)
Input pass(es)    : 1, processing 200 * 51.2 KiB chunks per pass
Read buffer size  : 4045.05 KiB * max 8 buffers
Hash method       : AVX512 (input), AVX512VL (recovery)

11th Gen Intel(R) Core(TM) i3-1115G4 @ 3.00GHz
  Multiply method : Affine (GFNI+AVX512) with 4480 B loop tiling, 4 threads
  Input batching  : 12 chunks, 2 batches
  Memory Usage    : 11.24 MiB (200 * 51.38 KiB chunks + 1233 KiB transfer buffer)

Finished          : 100.00%
Processing time   : 0.263 s
/ParPar/bin # ls -al
total 113860
drwxr-xr-x    1 root     root          4096 Aug 22 09:56 .
drwxr-xr-x    1 root     root          4096 Aug 22 09:28 ..
-rw-r--r--    1 root     root     104857600 Aug 22 09:54 blabla.bin
-rw-r--r--    1 root     root         40424 Aug 22 09:56 parfiles.par2
-rw-r--r--    1 root     root         92924 Aug 22 09:56 parfiles.vol000+001.par2
-rw-r--r--    1 root     root        145424 Aug 22 09:56 parfiles.vol001+002.par2
-rw-r--r--    1 root     root        290728 Aug 22 09:56 parfiles.vol003+004.par2
-rw-r--r--    1 root     root        541032 Aug 22 09:56 parfiles.vol007+008.par2
-rw-r--r--    1 root     root       1001336 Aug 22 09:56 parfiles.vol015+016.par2
-rw-r--r--    1 root     root       1881640 Aug 22 09:56 parfiles.vol031+032.par2
-rw-r--r--    1 root     root       3601944 Aug 22 09:56 parfiles.vol063+064.par2
-rw-r--r--    1 root     root       4074444 Aug 22 09:56 parfiles.vol127+073.par2
-rw-r--r--    1 root     root            26 Aug 22 09:26 parpar.cmd
-rwxr-xr-x    1 root     root         30842 Aug 22 09:26 parpar.js
/ParPar/bin #

animetosho · 2024-08-22T10:06:22Z

Thanks.

Could you try again using the dev branch instead of master?

sanderjo · 2024-08-22T10:30:35Z

build-parpar-riscv64.log
Ah, much better. TIL!

But: failure later on:

ar crs Release/obj.target/parpar_gf_c.a @Release/obj.target/parpar_gf_c.a.ar-file-list
  COPY Release/parpar_gf_c.a
  CXX(target) Release/obj.target/parpar_gf/src/gf.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_cpu.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl_init.o
  SOLINK_MODULE(target) Release/obj.target/parpar_gf.node
lto1: fatal error: target specific builtin not available
compilation terminated.
lto-wrapper: fatal error: g++ returned 1 exit status
compilation terminated.
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/../../../../riscv64-alpine-linux-musl/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make: *** [parpar_gf.target.mk:169: Release/obj.target/parpar_gf.node] Error 1
make: Leaving directory '/ParPar/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.<anonymous> (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:209:23)
gyp ERR! System Linux 6.1.15
gyp ERR! command "/usr/bin/node" "/usr/bin/node-gyp" "rebuild"
gyp ERR! cwd /ParPar
gyp ERR! node -v v20.15.1
gyp ERR! node-gyp -v v10.1.0
gyp ERR! not ok 
The command '/bin/sh -c git clone https://github.com/animetosho/ParPar.git && 	cd ParPar && 	git checkout dev && 	node-gyp rebuild && 	npm install' returned a non-zero code: 1

``

animetosho · 2024-08-22T10:54:41Z

Thanks again.

Hmm, can you try searching for and deleting , "PARPAR_LIBDL_SUPPORT" in binding.gyp and try again?

sanderjo · 2024-08-22T11:14:35Z

/ParPar # git checkout dev
branch 'dev' set up to track 'origin/dev'.
Switched to a new branch 'dev'
/ParPar # find . | grep binding.gyp
./binding.gyp
/ParPar # cat binding.gyp | grep PARPAR_LIBDL_SUPPORT
      "defines": ["NDEBUG", "PARPAR_LIBDL_SUPPORT"],
/ParPar # sed -i -e 's/, "PARPAR_LIBDL_SUPPORT"//g' binding.gyp
/ParPar #

OK, so let's put that into the Dockerfile

FROM alpine:edge


RUN apk update
RUN apk add nodejs npm python3 gcc g++ make automake autoconf git
RUN \
	git clone https://github.com/animetosho/ParPar.git && \
	cd ParPar && \
	git checkout dev && \
	sed -i -e 's/, "PARPAR_LIBDL_SUPPORT"//g' binding.gyp && \
	node-gyp rebuild && \
	npm install

EDIT: A grep shows PARPAR_LIBDL_SUPPORT is not there anymore in binding.gyp, so the sed is working

... still same error:

rm -f Release/obj.target/parpar_gf_c.a Release/obj.target/parpar_gf_c.a.ar-file-list; mkdir -p `dirname Release/obj.target/parpar_gf_c.a`
ar crs Release/obj.target/parpar_gf_c.a @Release/obj.target/parpar_gf_c.a.ar-file-list
  COPY Release/parpar_gf_c.a
  CXX(target) Release/obj.target/parpar_gf/src/gf.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_cpu.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl_init.o
  SOLINK_MODULE(target) Release/obj.target/parpar_gf.node
lto1: fatal error: target specific builtin not available
compilation terminated.
lto-wrapper: fatal error: g++ returned 1 exit status
compilation terminated.
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/../../../../riscv64-alpine-linux-musl/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make: *** [parpar_gf.target.mk:169: Release/obj.target/parpar_gf.node] Error 1
make: Leaving directory '/ParPar/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.<anonymous> (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:209:23)
gyp ERR! System Linux 6.1.15
gyp ERR! command "/usr/bin/node" "/usr/bin/node-gyp" "rebuild"
gyp ERR! cwd /ParPar
gyp ERR! node -v v20.15.1
gyp ERR! node-gyp -v v10.1.0
gyp ERR! not ok 
The command '/bin/sh -c git clone https://github.com/animetosho/ParPar.git && 	cd ParPar && 	git checkout dev && 	sed -i -e 's/, "PARPAR_LIBDL_SUPPORT"//g' binding.gyp && 	node-gyp rebuild && 	npm install' returned a non-zero code: 1
``

animetosho · 2024-08-22T12:33:46Z

I don't really know then. Maybe some issue with RVV and LTO?

Try replacing the node-gyp rebuild command with LDFLAGS=-fno-lto CFLAGS=-fno-lto CXXFLAGS=-fno-lto node-gyp rebuild to disable building with LTO.

sanderjo · 2024-08-22T12:56:42Z

I'll try.

In the meantime I had already something (likely?) ugly (based on nodejs/node@bec959e) , but still error


 sed -i -e 's/"-march=rv64gcv"/"-march=rv64gcv",  "-fno-lto"/g' binding.gyp
 sed -i -e 's/"-march=rv64gcv_zvbc1"/"-march=rv64gcv_zvbc1",  "-fno-lto"/g' binding.gyp


/ParPar # cat binding.gyp | grep  'fno-lto'
          "cflags": ["-mfpu=fp-armv8","-fno-lto"],
          "cxxflags": ["-mfpu=fp-armv8","-fno-lto"],
            "OTHER_CFLAGS": ["-mfpu=fp-armv8","-fno-lto"],
            "OTHER_CXXFLAGS": ["-mfpu=fp-armv8","-fno-lto"]
          "cflags": ["-mfpu=neon","-fno-lto"],
          "cxxflags": ["-mfpu=neon","-fno-lto"],
            "OTHER_CFLAGS": ["-mfpu=neon","-fno-lto"],
            "OTHER_CXXFLAGS": ["-mfpu=neon","-fno-lto"]
          "cflags": ["-mfpu=neon","-fno-lto"],
          "cxxflags": ["-mfpu=neon","-fno-lto"],
            "OTHER_CFLAGS": ["-mfpu=neon","-fno-lto"],
            "OTHER_CXXFLAGS": ["-mfpu=neon","-fno-lto"]
          "cflags": ["-mfpu=neon","-fno-lto"],
            "OTHER_CFLAGS": ["-mfpu=neon","-fno-lto"]
              "cflags": ["-march=rv64gcv",  "-fno-lto"],
              "cxxflags": ["-march=rv64gcv",  "-fno-lto"],
                "OTHER_CFLAGS": ["-march=rv64gcv",  "-fno-lto"],
                "OTHER_CXXFLAGS": ["-march=rv64gcv",  "-fno-lto"],
              "cflags": ["-march=rv64gcv_zvbc1",  "-fno-lto"],
              "cxxflags": ["-march=rv64gcv_zvbc1",  "-fno-lto"],
                "OTHER_CFLAGS": ["-march=rv64gcv_zvbc1",  "-fno-lto"],
                "OTHER_CXXFLAGS": ["-march=rv64gcv_zvbc1",  "-fno-lto"],

sanderjo · 2024-08-22T13:03:23Z

same error with LDFLAGS=-fno-lto CFLAGS=-fno-lto CXXFLAGS=-fno-lto node-gyp rebuild

ar crs Release/obj.target/parpar_gf_c.a @Release/obj.target/parpar_gf_c.a.ar-file-list
  COPY Release/parpar_gf_c.a
  CXX(target) Release/obj.target/parpar_gf/src/gf.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_cpu.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl_init.o
  SOLINK_MODULE(target) Release/obj.target/parpar_gf.node
lto1: fatal error: target specific builtin not available
compilation terminated.
lto-wrapper: fatal error: g++ returned 1 exit status
compilation terminated.
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/../../../../riscv64-alpine-linux-musl/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make: *** [parpar_gf.target.mk:169: Release/obj.target/parpar_gf.node] Error 1
make: Leaving directory '/ParPar/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.<anonymous> (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:209:23)
gyp ERR! System Linux 6.1.15
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /ParPar
gyp ERR! node -v v20.15.1
gyp ERR! node-gyp -v v10.1.0
gyp ERR! not ok
npm error code 1
npm error path /ParPar
npm error command failed
npm error command sh -c node-gyp rebuild
npm error A complete log of this run can be found in: /root/.npm/_logs/2024-08-22T12_54_27_970Z-debug-0.log
The command '/bin/sh -c git clone https://github.com/animetosho/ParPar.git &&   cd ParPar &&    git checkout dev &&     grep PARPAR_LIBDL_SUPPORT binding.gyp | wc -l &&       sed -i -e 's/, "PARPAR_LIBDL_SUPPORT"//g' binding.gyp &&        grep PARPAR_LIBDL_SUPPORT binding.gyp | wc -l  &&       LDFLAGS=-fno-lto CFLAGS=-fno-lto CXXFLAGS=-fno-lto node-gyp rebuild &&         npm install' returned a non-zero code: 1

sanderjo · 2024-08-22T13:35:45Z

build_parpar-ubuntu2410-riscv64.log
Progress?

I built from Ubuntu:24.10 (also with gcc-14.2.0, still on RISCV64), and that works. See tail below.

With very clean Dockerfile (no specials):

FROM ubuntu:24.10

RUN apt update

RUN apt install -y nodejs npm python3 gcc g++ make automake autoconf git

RUN \
	git clone https://github.com/animetosho/ParPar.git && \
	cd ParPar && \
	git checkout dev && \
	node-gyp rebuild && \
	npm install

Which binary can I check for rvv commands (with objdump)? I only see .js and . json files, and there must be a binary?

root@78389c30ed8a:/ParPar/bin# fallocate -l100M blabla.bin
root@78389c30ed8a:/ParPar/bin# node parpar.js  -s2000 --min-input-slices=1 -r10% -d pow2 -o parfiles *bin
Input data        : 100 MiB (2000 slices from 1 file)
Recovery data     : 10 MiB (200 * 51.2 KiB slices)
Input pass(es)    : 1, processing 200 * 51.2 KiB chunks per pass
Read buffer size  : 4045.05 KiB * max 8 buffers
Hash method       : Scalar+Generic (input), Scalar (recovery)

Spacemit(R) X60
  Multiply method : Shuffle-128 (RVV) with 17.13 KiB loop tiling, 8 threads
  Input batching  : 12 chunks, 2 batches
  Memory Usage    : 11.22 MiB (200 * 51.31 KiB chunks + 1231.5 KiB transfer buffer)

Finished          : 100.00%
Processing time   : 4.256 s

rm -f Release/obj.target/hasher_sse2.a Release/obj.target/hasher_sse2.a.ar-file-list; mkdir -p `dirname Release/obj.target/hasher_sse2.a`
ar crs Release/obj.target/hasher_sse2.a @Release/obj.target/hasher_sse2.a.ar-file-list
  COPY Release/hasher_sse2.a
  CXX(target) Release/obj.target/hasher_sve2/hasher/hasher_sve2.o
rm -f Release/obj.target/hasher_sve2.a Release/obj.target/hasher_sve2.a.ar-file-list; mkdir -p `dirname Release/obj.target/hasher_sve2.a`
ar crs Release/obj.target/hasher_sve2.a @Release/obj.target/hasher_sve2.a.ar-file-list
  COPY Release/hasher_sve2.a
  CXX(target) Release/obj.target/hasher_xop/hasher/hasher_xop.o
rm -f Release/obj.target/hasher_xop.a Release/obj.target/hasher_xop.a.ar-file-list; mkdir -p `dirname Release/obj.target/hasher_xop.a`
ar crs Release/obj.target/hasher_xop.a @Release/obj.target/hasher_xop.a.ar-file-list
  COPY Release/hasher_xop.a
  CC(target) Release/obj.target/parpar_gf_c/gf16/opencl-include/cl.o
  CC(target) Release/obj.target/parpar_gf_c/gf16/gfmat_coeff.o
rm -f Release/obj.target/parpar_gf_c.a Release/obj.target/parpar_gf_c.a.ar-file-list; mkdir -p `dirname Release/obj.target/parpar_gf_c.a`
ar crs Release/obj.target/parpar_gf_c.a @Release/obj.target/parpar_gf_c.a.ar-file-list
  COPY Release/parpar_gf_c.a
  CXX(target) Release/obj.target/parpar_gf/src/gf.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_cpu.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl_init.o
  SOLINK_MODULE(target) Release/obj.target/parpar_gf.node
  COPY Release/parpar_gf.node
make: Leaving directory '/ParPar/build'
gyp info ok

added 2 packages, and audited 3 packages in 5m

found 0 vulnerabilities
 ---> Removed intermediate container 0405e39b3f7b
 ---> 946014a52d12
Successfully built 946014a52d12

animetosho · 2024-08-23T11:22:50Z

Hmm, maybe node-gyp is overriding the CFLAGS.
I really don't know, so I can only offer 'shot-in-the-dark' suggestions:
Does replacing node-gyp rebuild with node-gyp rebuild --enable_lto=false do anything?

Progress?

My guess is it's something with the musl build chain that doesn't like it. I think you previously built this fine on Ubuntu.

Which binary can I check for rvv commands (with objdump)?

The library should be build/Release/parpar_gf.node

Your output shows Multiply method : Shuffle-128 (RVV) though, so RVV is compiled in.

sanderjo · 2024-08-24T08:08:56Z

Does replacing node-gyp rebuild with node-gyp rebuild --enable_lto=false do anything?

Same error.

Shall I close this issue as "Alpine specific"?

lto1: fatal error: target specific builtin not available
compilation terminated.
lto-wrapper: fatal error: g++ returned 1 exit status
compilation terminated.
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/../../../../riscv64-alpine-linux-musl/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make: *** [parpar_gf.target.mk:169: Release/obj.target/parpar_gf.node] Error 1

sanderjo · 2024-08-24T08:10:30Z

In the Ubuntu build version indeed rvv commands inside /ParPar/build/Release/parpar_gf.node

Cool

root@b5a1c081e006:~# objdump -d /ParPar/build/Release/parpar_gf.node | awk '{ print $3 }' | sort -u | grep -e ^v | head
vadd.vv
vand.vi
vand.vv
vfirst.m
vl1r.v
vl1re16.v
vle8.v
vlm.v
vlseg2e16.v
vlseg2e8.v

EDIT

I could have found it:

/ParPar# find . -type f -executable -print | grep -vi -e js$ -e sample$ -e sh$
./build/Release/parpar_gf.node
./build/Release/obj.target/parpar_gf.node

Thanks

animetosho · 2024-08-24T11:07:34Z

Shall I close this issue as "Alpine specific"?

I'm willing to diagnose this as long as you're willing to test out my random shots in the dark. I don't really have your environment set up, so it's hard for me to replicate.

If you are willing to diagnose further:
Can you try a grep lto build/* to see if there's any LTO refs in the build scripts?
Also a node-gyp rebuild -v to show all the compilation commands might yield some more info.

It's odd that it looks like LTO is being invoked despite attempts to disable it.

sanderjo · 2024-08-25T08:48:22Z

I'm willing to diagnose this as long as you're willing to test out my random shots in the dark.

Thank you.

I don't really have your environment set up, so it's hard for me to replicate.

Still my offer to you: via IPv6 access to my riscv system

I did stuff, and I'm dumping it here, without thinking about it too much
build-parpar-riscv64---2024-08-25---no-lto---verbose-build.log

TLDR: it now builds (maybe because of ignoring the returncode in the Dockerfile). EDIT: After a manual npm install async I can run parpar.

Docker build command longer, but nothing special:

sander@k1:~/parpar-docker$ docker build -t no-lto---verbose-build . -f Dockerfile.no-lto---verbose-build 2>&1 | tee build-parpar-riscv64---2024-08-25---no-lto---verbose-build.log

Dockerfile now with all precautions, plus ignoring the output of the build

$ cat Dockerfile.no-lto---verbose-build
FROM alpine:edge


RUN apk update
RUN apk add nodejs npm python3 gcc g++ make automake autoconf git
RUN \
	git clone https://github.com/animetosho/ParPar.git && \
	cd ParPar && \
	git checkout dev && \
	grep PARPAR_LIBDL_SUPPORT binding.gyp | wc -l && \
	sed -i -e 's/, "PARPAR_LIBDL_SUPPORT"//g' binding.gyp && \
	grep PARPAR_LIBDL_SUPPORT binding.gyp | wc -l  && \
	LDFLAGS=-fno-lto CFLAGS=-fno-lto CXXFLAGS=-fno-lto node-gyp rebuild -v ; exit 0 \
	npm install
[build-parpar-riscv64---2024-08-25---no-lto---verbose-build.log](https://github.com/user-attachments/files/16739535/build-parpar-riscv64---2024-08-25---no-lto---verbose-build.log)

EDIT: the below solved with npm install async

/ParPar/bin # fallocate -l100M blabla.bin
/ParPar/bin # 
/ParPar/bin # node parpar.js  -s2000 --min-input-slices=1 -r10% -d pow2 -o parfiles *bin
node:internal/modules/cjs/loader:1148
  throw err;
  ^

Error: Cannot find module 'async'
Require stack:
- /ParPar/lib/par2.js
- /ParPar/lib/parpar.js
- /ParPar/bin/parpar.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1145:15)
    at Module._load (node:internal/modules/cjs/loader:986:27)
    at Module.require (node:internal/modules/cjs/loader:1233:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/ParPar/lib/par2.js:5:13)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Module._load (node:internal/modules/cjs/loader:1024:12)
    at Module.require (node:internal/modules/cjs/loader:1233:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/ParPar/lib/par2.js',
    '/ParPar/lib/parpar.js',
    '/ParPar/bin/parpar.js'
  ]
}

Node.js v20.15.1

  g++ -o Release/obj.target/parpar_gf.node -shared -pthread -rdynamic -flto=4 -fuse-linker-plugin -ffat-lto-objects  -fno-lto -Wl,-soname=parpar_gf.node -Wl,--start-group Release/obj.target/parpar_gf/src/gf.o Release/obj.target/parpar_gf/gf16/controller.o Release/obj.target/parpar_gf/gf16/controller_cpu.o Release/obj.target/parpar_gf/gf16/controller_ocl.o Release/obj.target/parpar_gf/gf16/controller_ocl_init.o Release/obj.target/parpar_gf_c.a Release/obj.target/gf16.a Release/obj.target/gf16_generic.a Release/obj.target/gf16_sse2.a Release/obj.target/gf16_ssse3.a Release/obj.target/gf16_avx.a Release/obj.target/gf16_avx2.a Release/obj.target/gf16_avx512.a Release/obj.target/gf16_vbmi.a Release/obj.target/gf16_gfni.a Release/obj.target/gf16_gfni_avx2.a Release/obj.target/gf16_gfni_avx512.a Release/obj.target/gf16_gfni_avx10.a Release/obj.target/gf16_neon.a Release/obj.target/gf16_sha3.a Release/obj.target/gf16_sve.a Release/obj.target/gf16_sve2.a Release/obj.target/gf16_rvv.a Release/obj.target/gf16_rvv_zvbc.a Release/obj.target/hasher.a Release/obj.target/hasher_sse2.a Release/obj.target/hasher_clmul.a Release/obj.target/hasher_xop.a Release/obj.target/hasher_bmi1.a Release/obj.target/hasher_avx2.a Release/obj.target/hasher_avx512.a Release/obj.target/hasher_avx512vl.a Release/obj.target/hasher_armcrc.a Release/obj.target/hasher_neon.a Release/obj.target/hasher_neoncrc.a Release/obj.target/hasher_sve2.a Release/obj.target/hasher_rvzbc.a Release/obj.target/hasher_c.a -Wl,--end-group 
  ln -f "Release/obj.target/parpar_gf.node" "Release/parpar_gf.node" 2>/dev/null || (rm -rf "Release/parpar_gf.node" && cp -af "Release/obj.target/parpar_gf.node" "Release/parpar_gf.node")
make: Leaving directory '/ParPar/build'
gyp info ok 
 ---> Removed intermediate container 8baf2d0c2ba8
 ---> 56f80b5dc1b7
Successfully built 56f80b5dc1b7
Successfully tagged no-lto---verbose-build:latest
sander@k1:~/parpar-docker$

Binaries are there

/ParPar # find . -type f -executable -print | grep -vi -e js$ -e sample$ -e sh$
./build/Release/parpar_gf.node
./build/Release/obj.target/parpar_gf.node

/ParPar # grep lto build/*
build/config.gypi:    "enable_lto": "true",
build/gf16.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_avx.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_avx.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_avx2.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_avx2.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_avx512.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_avx512.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_generic.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_generic.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_gfni.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_gfni.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_gfni_avx10.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_gfni_avx10.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_gfni_avx2.target.mk:	 -flto=4 -fuse-linker-plugin -ffat-lto-objects

animetosho · 2024-08-26T06:45:49Z

Thanks, looks like LTO is still being enabled.

Can you try with this Dockerfile instead?

FROM alpine:edge


RUN apk update
RUN apk add nodejs npm python3 gcc g++ make automake autoconf git
RUN \
	git clone https://github.com/animetosho/ParPar.git && \
	cd ParPar && \
	git checkout dev && \
	node-gyp rebuild --enable_lto=false -v && \
	grep lto build/* ; exit 0 \
	npm install

sanderjo · 2024-08-26T07:52:04Z

That one built without problems.

build.Dockerfile.animetosho_2024-08-26.log

Still strange: an error when trying to run parpar.js ... which is solved by manuall installing async. Which I tried putting into Dockerfile, without solving it.

sander@k1:~/parpar-docker$ docker run -it parpar-animetosho_2024-08-26 sh
/ # cd ParPar/bin

/ParPar/bin # node parpar.js --help 
node:internal/modules/cjs/loader:1148
  throw err;
  ^

Error: Cannot find module 'async'
Require stack:
- /ParPar/lib/par2.js
- /ParPar/lib/parpar.js
- /ParPar/bin/parpar.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1145:15)
    at Module._load (node:internal/modules/cjs/loader:986:27)
    at Module.require (node:internal/modules/cjs/loader:1233:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/ParPar/lib/par2.js:5:13)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Module._load (node:internal/modules/cjs/loader:1024:12)
    at Module.require (node:internal/modules/cjs/loader:1233:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/ParPar/lib/par2.js',
    '/ParPar/lib/parpar.js',
    '/ParPar/bin/parpar.js'
  ]
}

Node.js v20.15.1

/ParPar/bin # npm install async

added 2 packages, and audited 3 packages in 9s

found 0 vulnerabilities
/ParPar/bin #

/ParPar/bin # node parpar.js --help 2>&1 | head -5
ParPar v0.4.4-dev
High performance PAR2 creation tool
-----------------------------------

Usage: parpar -s <slice_size/count> -o <output> [options] [--] <input1> [<input2>...]
/ParPar/bin #

Next: building without the hack exit 0, and with the &&

The grep output:

build/config.gypi:    "enable_lto": "false",

  g++ -o Release/obj.target/parpar_gf.node -shared -pthread -rdynamic  -Wl,-soname=parpar_gf.node -Wl,--start-group Release/obj.target/parpar_gf/src/gf.o Release/obj.target/parpar_gf/gf16/controller.o Release/obj.target/parpar_gf/gf16/controller_cpu.o Release/obj.target/parpar_gf/gf16/controller_ocl.o Release/obj.target/parpar_gf/gf16/controller_ocl_init.o Release/obj.target/parpar_gf_c.a Release/obj.target/gf16.a Release/obj.target/gf16_generic.a Release/obj.target/gf16_sse2.a Release/obj.target/gf16_ssse3.a Release/obj.target/gf16_avx.a Release/obj.target/gf16_avx2.a Release/obj.target/gf16_avx512.a Release/obj.target/gf16_vbmi.a Release/obj.target/gf16_gfni.a Release/obj.target/gf16_gfni_avx2.a Release/obj.target/gf16_gfni_avx512.a Release/obj.target/gf16_gfni_avx10.a Release/obj.target/gf16_neon.a Release/obj.target/gf16_sha3.a Release/obj.target/gf16_sve.a Release/obj.target/gf16_sve2.a Release/obj.target/gf16_rvv.a Release/obj.target/gf16_rvv_zvbc.a Release/obj.target/hasher.a Release/obj.target/hasher_sse2.a Release/obj.target/hasher_clmul.a Release/obj.target/hasher_xop.a Release/obj.target/hasher_bmi1.a Release/obj.target/hasher_avx2.a Release/obj.target/hasher_avx512.a Release/obj.target/hasher_avx512vl.a Release/obj.target/hasher_armcrc.a Release/obj.target/hasher_neon.a Release/obj.target/hasher_neoncrc.a Release/obj.target/hasher_sve2.a Release/obj.target/hasher_rvzbc.a Release/obj.target/hasher_c.a -Wl,--end-group 
  ln -f "Release/obj.target/parpar_gf.node" "Release/parpar_gf.node" 2>/dev/null || (rm -rf "Release/parpar_gf.node" && cp -af "Release/obj.target/parpar_gf.node" "Release/parpar_gf.node")
make: Leaving directory '/ParPar/build'
gyp info ok 
build/config.gypi:    "enable_lto": "false",
 ---> Removed intermediate container e7011cc871f2
 ---> 4076e6b9233a
Successfully built 4076e6b9233a
Successfully tagged parpar-animetosho_2024-08-26:latest

real	5m25.475s
user	0m0.512s
sys	0m0.574s

animetosho · 2024-08-26T10:47:15Z

That one built without problems.

Yay!

Still strange: an error when trying to run parpar.js ... which is solved by manuall installing async

Usually the npm install will install the dependency.

sanderjo changed the title ~~RISCV64: ../gf16/gf16_rvv_common.h:33:20: error: incompatible type for argument 3 of '__riscv_vxor_vx_i16m1_m'~~ RISCV64 on Alpine (not on Ubuntu): ../gf16/gf16_rvv_common.h:33:20: error: incompatible type for argument 3 of '__riscv_vxor_vx_i16m1_m' Aug 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RISCV64 on Alpine (not on Ubuntu): ../gf16/gf16_rvv_common.h:33:20: error: incompatible type for argument 3 of '__riscv_vxor_vx_i16m1_m' #62

RISCV64 on Alpine (not on Ubuntu): ../gf16/gf16_rvv_common.h:33:20: error: incompatible type for argument 3 of '__riscv_vxor_vx_i16m1_m' #62

sanderjo commented Aug 22, 2024 •

edited

Loading

sanderjo commented Aug 22, 2024

sanderjo commented Aug 22, 2024

animetosho commented Aug 22, 2024

sanderjo commented Aug 22, 2024 •

edited

Loading

animetosho commented Aug 22, 2024

sanderjo commented Aug 22, 2024 •

edited

Loading

animetosho commented Aug 22, 2024 •

edited

Loading

sanderjo commented Aug 22, 2024

sanderjo commented Aug 22, 2024

sanderjo commented Aug 22, 2024 •

edited

Loading

animetosho commented Aug 23, 2024

sanderjo commented Aug 24, 2024

sanderjo commented Aug 24, 2024 •

edited

Loading

animetosho commented Aug 24, 2024

sanderjo commented Aug 25, 2024 •

edited

Loading

animetosho commented Aug 26, 2024

sanderjo commented Aug 26, 2024

animetosho commented Aug 26, 2024

RISCV64 on Alpine (not on Ubuntu): ../gf16/gf16_rvv_common.h:33:20: error: incompatible type for argument 3 of '__riscv_vxor_vx_i16m1_m' #62

RISCV64 on Alpine (not on Ubuntu): ../gf16/gf16_rvv_common.h:33:20: error: incompatible type for argument 3 of '__riscv_vxor_vx_i16m1_m' #62

Comments

sanderjo commented Aug 22, 2024 • edited Loading

sanderjo commented Aug 22, 2024

sanderjo commented Aug 22, 2024

animetosho commented Aug 22, 2024

sanderjo commented Aug 22, 2024 • edited Loading

animetosho commented Aug 22, 2024

sanderjo commented Aug 22, 2024 • edited Loading

animetosho commented Aug 22, 2024 • edited Loading

sanderjo commented Aug 22, 2024

sanderjo commented Aug 22, 2024

sanderjo commented Aug 22, 2024 • edited Loading

animetosho commented Aug 23, 2024

sanderjo commented Aug 24, 2024

sanderjo commented Aug 24, 2024 • edited Loading

animetosho commented Aug 24, 2024

sanderjo commented Aug 25, 2024 • edited Loading

animetosho commented Aug 26, 2024

sanderjo commented Aug 26, 2024

animetosho commented Aug 26, 2024

sanderjo commented Aug 22, 2024 •

edited

Loading

sanderjo commented Aug 22, 2024 •

edited

Loading

sanderjo commented Aug 22, 2024 •

edited

Loading

animetosho commented Aug 22, 2024 •

edited

Loading

sanderjo commented Aug 22, 2024 •

edited

Loading

sanderjo commented Aug 24, 2024 •

edited

Loading

sanderjo commented Aug 25, 2024 •

edited

Loading