yaml.h not found and psych.rb:455:in parse_stream: undefined method parse for #<Psych::Parser

[dorianmariecom]

Hi,

I'm getting two different errors on different environments (both Linux) when upgrading to Psych 5 but it works fine on macOS.

First error is:

I'm getting this output when upgrading psych from 4 to 5:

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
494 |  
495 | current directory: /usr/local/bundle/gems/skylight-5.3.4/ext
496 | /usr/local/bin/ruby -I /usr/local/lib/ruby/3.1.0 -r
497 | ./siteconf20221205-17-o7zxoa.rb extconf.rb
498 | musl libc (x86_64)
499 | Version 1.2.3
500 | Dynamic Program Loader
501 | Usage: /lib/ld-musl-x86_64.so.1 [options] [--] pathname
502 | *** extconf.rb failed ***
503 | Could not create Makefile due to some reason, probably lack of necessary
504 | libraries and/or headers.  Check the mkmf.log file for more details.  You may
505 | need configuration options.
506 |  
507 | Provided configuration options:
508 | --with-opt-dir
509 | --without-opt-dir
510 | --with-opt-include
511 | --without-opt-include=${opt-dir}/include
512 | --with-opt-lib
513 | --without-opt-lib=${opt-dir}/lib
514 | --with-make-prog
515 | --without-make-prog
516 | --srcdir=.
517 | --curdir
518 | --ruby=/usr/local/bin/$(RUBY_BASE_NAME)
519 | /usr/local/lib/ruby/3.1.0/psych.rb:455:in `parse_stream': undefined method
520 | `parse' for #<Psych::Parser:0x00007f870c732950
521 | @handler=#<Psych::Handlers::DocumentStream:0x00007f870c732b58 @stack=[],
522 | @last=nil, @root=nil, @start_line=nil, @start_column=nil, @end_line=nil,
523 | @end_column=nil, @block=#<Proc:0x00007f870c7329c8
524 | /usr/local/lib/ruby/3.1.0/psych.rb:399>>, @external_encoding=0> (NoMethodError)
525 |  
526 | parser.parse yaml, filename
527 | ^^^^^^
528 | from /usr/local/lib/ruby/3.1.0/psych.rb:399:in `parse'
529 | from /usr/local/lib/ruby/3.1.0/psych.rb:324:in `safe_load'
530 | from /usr/local/lib/ruby/3.1.0/psych.rb:370:in `load'
531 | from /usr/local/lib/ruby/3.1.0/psych.rb:671:in `block in load_file'
532 | from /usr/local/lib/ruby/3.1.0/psych.rb:670:in `open'
533 | from /usr/local/lib/ruby/3.1.0/psych.rb:670:in `load_file'
534 | from extconf.rb:121:in `<main>'
535 | I, [2022-12-05T09:14:41.592907 #3150]  INFO -- :
536 | SKYLIGHT_HDR_PATH=/usr/local/bundle/gems/skylight-5.3.4/ext;
537 | SKYLIGHT_LIB_PATH=/usr/local/bundle/gems/skylight-5.3.4/lib/skylight/native/x86_64-linux-musl
538

And second error is:

  Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
  
  current directory:
  /home/runner/work/albaik-web/albaik-web/vendor/bundle/ruby/3.1.0/gems/psych-5.0.0/ext/psych
  /opt/hostedtoolcache/Ruby/3.1.2/x64/bin/ruby -I
  /opt/hostedtoolcache/Ruby/3.1.2/x64/lib/ruby/3.1.0 -r
  ./siteconf20221205-2458-sr1mb3.rb extconf.rb
  checking for yaml.h... no
  yaml.h not found
  *** extconf.rb failed ***
  Could not create Makefile due to some reason, probably lack of necessary
  libraries and/or headers.  Check the mkmf.log file for more details.  You may
  need configuration options.
  
  Provided configuration options:
  	--with-opt-dir
  	--without-opt-dir
  	--with-opt-include
  	--without-opt-include=${opt-dir}/include
  	--with-opt-lib
  	--without-opt-lib=${opt-dir}/lib
  	--with-make-prog
  	--without-make-prog
  	--srcdir=.
  	--curdir
  	--ruby=/opt/hostedtoolcache/Ruby/3.1.2/x64/bin/$(RUBY_BASE_NAME)
  	--with-libyaml-source-dir
  	--without-libyaml-source-dir
  	--with-yaml-0.1-dir
  	--without-yaml-0.1-dir
  	--with-yaml-0.1-include
  	--without-yaml-0.1-include=${yaml-0.1-dir}/include
  	--with-yaml-0.1-lib
  	--without-yaml-0.1-lib=${yaml-0.1-dir}/lib
  	--with-yaml-0.1-config
  	--without-yaml-0.1-config
  	--with-pkg-config
  	--without-pkg-config
  	--with-libyaml-dir
  	--without-libyaml-dir
  	--with-libyaml-include
  	--without-libyaml-include=${libyaml-dir}/include
  	--with-libyaml-lib
  	--without-libyaml-lib=${libyaml-dir}/lib

See skylightio/skylight-ruby#437

[sebbASF]

I'm seeing undefined method `_native_parse' for #Psych::Parser:0x......

[hosamaly]

Could this be related to #541? Is libyaml-dev installed? Is it up to date (i.e. v0.2.5)?

[kbarrette]

I'm also seeing problems with missing libyaml - the Ubuntu image that dependabot uses doesn't seem to include libyaml, which makes psych 5.0.0 install fail, which means dependabot can't check my dependencies for projects that use latest psych.

Not quite sure what needs to be fixed: psych, dependabot, or the image?

[bryansoto]

Ran into the "yaml.h not found" error with the RubyInstaller for Windows which uses MINGW. For others in the same boat, you will need to run pacman -S libyaml-devel mingw-w64-ucrt-x86_64-libyaml in a MINGW prompt. After that, the upgrade to psych 5 succeeded for me.

[dorianmariecom]

Could this be related to #541? Is libyaml-dev installed? Is it up to date (i.e. v0.2.5)?

With libyaml-dev installed I get the same error on AWS and on GitHub Actions (related to the Skylight gem)

[kbarrette]

Anyone like me who got here via a failing github action may be interested in ruby/setup-ruby#409 and actions/runner-images#6725

[MSP-Greg]

@bryansoto

JFYI, on Windows, I think pacman -S mingw-w64-ucrt-x86_64-libyaml will suffice, as libyaml-devel is an MSYS package.

Quite a few packages on Ubuntu are split into 'runtime' packages (the standard one), and a 'dev' package. With MSYS2 and vcpkg, they are often a single package...

[dometto]

I'm still seeing the undefined method parse issue on Github Actions with the ubuntu-latest image, even after the addition of libyaml-dev in actions/runner-images#6725, but only on ruby 3.0.0. Is this a bug or am I missing another dependency?

[tindron]

I'm seeing this with ubuntu-20.04 and ubuntu-latest and ruby 3.1.3. I got around it by removing rdoc to get rid of the psych dependency.

[HarlemSquirrel]

I ran into this problem with psych 5.0 on CircleCI running Ubuntu 22.04 with libyaml-dev 0.2.2-1build2. I downgraded to psych 4.0 for now.

CI output

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /home/circleci/project/vendor/bundle/ruby/3.1.0/gems/nokogiri-1.13.10/ext/nokogiri
/usr/local/bin/ruby -I /usr/local/lib/ruby/3.1.0 extconf.rb
checking for whether -std=c99 is accepted as CFLAGS... yes
checking for whether -Wno-declaration-after-statement is accepted as CFLAGS... yes
checking for whether -g is accepted as CFLAGS... yes
checking for whether -Winline is accepted as CFLAGS... yes
checking for whether -Wmissing-noreturn is accepted as CFLAGS... yes
Building nokogiri using packaged libraries.
Static linking is enabled.
Cross build is disabled.
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/usr/local/bin/$(RUBY_BASE_NAME)
	--help
	--clean
	--prevent-strip
	--enable-system-libraries
	--disable-system-libraries
	--use-system-libraries
	--enable-system-libraries
	--disable-system-libraries
	--use-system-libraries
	--enable-static
	--disable-static
	--enable-cross-build
	--disable-cross-build
/usr/local/lib/ruby/3.1.0/psych.rb:455:in `parse_stream': undefined method `parse' for #<Psych::Parser:0x00007fc4d7385358
@handler=#<Psych::Handlers::DocumentStream:0x00007fc4d73854e8 @stack=[], @last=nil, @root=nil, @start_line=nil,
@start_column=nil, @end_line=nil, @end_column=nil, @block=#<Proc:0x00007fc4d73853d0 /usr/local/lib/ruby/3.1.0/psych.rb:399>>,
@external_encoding=0> (NoMethodError)

      parser.parse yaml, filename
            ^^^^^^
	from /usr/local/lib/ruby/3.1.0/psych.rb:399:in `parse'
	from /usr/local/lib/ruby/3.1.0/psych.rb:323:in `safe_load'
	from /usr/local/lib/ruby/3.1.0/psych.rb:369:in `load'
	from /usr/local/lib/ruby/3.1.0/psych.rb:671:in `block in load_file'
	from /usr/local/lib/ruby/3.1.0/psych.rb:670:in `open'
	from /usr/local/lib/ruby/3.1.0/psych.rb:670:in `load_file'
	from extconf.rb:670:in `<main>'

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /home/circleci/project/vendor/bundle/ruby/3.1.0/extensions/x86_64-linux/3.1.0/nokogiri-1.13.10/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /home/circleci/project/vendor/bundle/ruby/3.1.0/gems/nokogiri-1.13.10 for inspection.
Results logged to /home/circleci/project/vendor/bundle/ruby/3.1.0/extensions/x86_64-linux/3.1.0/nokogiri-1.13.10/gem_make.out

  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:102:in `run'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:28:in `build'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:171:in `build_extension'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:205:in `block in build_extensions'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:202:in `each'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:202:in `build_extensions'
  /usr/local/lib/ruby/3.1.0/rubygems/installer.rb:843:in `build_extensions'
/home/circleci/project/vendor/bundle/ruby/3.1.0/gems/bundler-2.4.3/lib/bundler/rubygems_gem_installer.rb:72:in
`build_extensions'
  /home/circleci/project/vendor/bundle/ruby/3.1.0/gems/bundler-2.4.3/lib/bundler/rubygems_gem_installer.rb:28:in `install'
  /home/circleci/project/vendor/bundle/ruby/3.1.0/gems/bundler-2.4.3/lib/bundler/source/rubygems.rb:200:in `install'
  /home/circleci/project/vendor/bundle/ruby/3.1.0/gems/bundler-2.4.3/lib/bundler/installer/gem_installer.rb:54:in `install'
/home/circleci/project/vendor/bundle/ruby/3.1.0/gems/bundler-2.4.3/lib/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/home/circleci/project/vendor/bundle/ruby/3.1.0/gems/bundler-2.4.3/lib/bundler/installer/parallel_installer.rb:155:in
`do_install'
/home/circleci/project/vendor/bundle/ruby/3.1.0/gems/bundler-2.4.3/lib/bundler/installer/parallel_installer.rb:146:in `block
in worker_pool'
  /home/circleci/project/vendor/bundle/ruby/3.1.0/gems/bundler-2.4.3/lib/bundler/worker.rb:62:in `apply_func'
  /home/circleci/project/vendor/bundle/ruby/3.1.0/gems/bundler-2.4.3/lib/bundler/worker.rb:57:in `block in process_queue'
  /home/circleci/project/vendor/bundle/ruby/3.1.0/gems/bundler-2.4.3/lib/bundler/worker.rb:54:in `loop'
  /home/circleci/project/vendor/bundle/ruby/3.1.0/gems/bundler-2.4.3/lib/bundler/worker.rb:54:in `process_queue'
/home/circleci/project/vendor/bundle/ruby/3.1.0/gems/bundler-2.4.3/lib/bundler/worker.rb:90:in `block (2 levels) in
create_threads'

An error occurred while installing nokogiri (1.13.10), and Bundler cannot continue.

In Gemfile:
  bootstrap-multiselect-rails was resolved to 0.9.9, which depends on
    rails was resolved to 7.0.4, which depends on
      actioncable was resolved to 7.0.4, which depends on
        actionpack was resolved to 7.0.4, which depends on
          actionview was resolved to 7.0.4, which depends on
            rails-dom-testing was resolved to 2.0.3, which depends on
              nokogiri

Not sure why but I don't run into this issue on my Pop!_OS 22.04 laptop running the same version of libyaml-dev.

[overtube]

gem_make.out

/home/ui/.rbenv/versions/3.1.2/bin/ruby -I /home/ui/.rbenv/versions/3.1.2/lib/ruby/3.1.0 -r ./siteconf20230623-28076-xc2we6.rb extconf.rb
checking for -lpthread... yes
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/home/ui/.rbenv/versions/3.1.2/bin/$(RUBY_BASE_NAME)
	--with-pthread-dir
	--without-pthread-dir
	--with-pthread-include
	--without-pthread-include=${pthread-dir}/include
	--with-pthread-lib
	--without-pthread-lib=${pthread-dir}/lib
	--with-pthreadlib
	--without-pthreadlib
	--enable-debug
	--disable-debug
	--enable-asan
	--disable-asan
/home/ui/.rbenv/versions/3.1.2/lib/ruby/3.1.0/psych.rb:455:in `parse_stream': undefined method `parse' for #<Psych::Parser:0x00007f6b17bda1d0 @handler=#<Psych::Handlers::DocumentStream:0x00007f6b17bda3b0 @stack=[], @last=nil, @root=nil, @start_line=nil, @start_column=nil, @end_line=nil, @end_column=nil, @block=#<Proc:0x00007f6b17bda248 /home/ui/.rbenv/versions/3.1.2/lib/ruby/3.1.0/psych.rb:399>>, @external_encoding=0> (NoMethodError)

      parser.parse yaml, filename
            ^^^^^^
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/3.1.0/psych.rb:399:in `parse'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/3.1.0/psych.rb:272:in `unsafe_load'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/libv8-node-18.16.0.0-x86_64-linux/ext/libv8-node/location.rb:21:in `block in load!'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/libv8-node-18.16.0.0-x86_64-linux/ext/libv8-node/location.rb:20:in `open'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/libv8-node-18.16.0.0-x86_64-linux/ext/libv8-node/location.rb:20:in `load!'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/libv8-node-18.16.0.0-x86_64-linux/lib/libv8/node.rb:8:in `configure_makefile'
	from extconf.rb:77:in `<main>'

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/extensions/x86_64-linux/3.1.0/mini_racer-0.8.0/mkmf.log

extconf failed, exit code 1

mkmf.log

LD_LIBRARY_PATH=.:/home/ui/.rbenv/versions/3.1.2/lib:/usr/local/lib "gcc -o conftest -I/home/ui/.rbenv/versions/3.1.2/include/ruby-3.1.0/x86_64-linux -I/home/ui/.rbenv/versions/3.1.2/include/ruby-3.1.0/ruby/backward -I/home/ui/.rbenv/versions/3.1.2/include/ruby-3.1.0 -I. -I/home/ui/.rbenv/versions/3.1.2/include    -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wdeprecated-declarations -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wwrite-strings -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable -Wundef  -fPIC conftest.c  -L. -L/home/ui/.rbenv/versions/3.1.2/lib -Wl,-rpath,/home/ui/.rbenv/versions/3.1.2/lib -L. -L/home/ui/.rbenv/versions/3.1.2/lib  -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,--no-as-needed     -Wl,-rpath,/home/ui/.rbenv/versions/3.1.2/lib -L/home/ui/.rbenv/versions/3.1.2/lib -lruby  -lm  -lc"
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: int main(int argc, char **argv)
4: {
5:   return !!argv[argc];
6: }
/* end */

LD_LIBRARY_PATH=.:/home/ui/.rbenv/versions/3.1.2/lib:/usr/local/lib "gcc -o conftest -I/home/ui/.rbenv/versions/3.1.2/include/ruby-3.1.0/x86_64-linux -I/home/ui/.rbenv/versions/3.1.2/include/ruby-3.1.0/ruby/backward -I/home/ui/.rbenv/versions/3.1.2/include/ruby-3.1.0 -I. -I/home/ui/.rbenv/versions/3.1.2/include    -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wdeprecated-declarations -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wwrite-strings -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable -Wundef  -fPIC conftest.c  -L. -L/home/ui/.rbenv/versions/3.1.2/lib -Wl,-rpath,/home/ui/.rbenv/versions/3.1.2/lib -L. -L/home/ui/.rbenv/versions/3.1.2/lib  -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,--no-as-needed     -Wl,-rpath,/home/ui/.rbenv/versions/3.1.2/lib -L/home/ui/.rbenv/versions/3.1.2/lib -lruby -lpthread  -lm  -lc"
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: /*top*/
 4: extern int t(void);
 5: int main(int argc, char **argv)
 6: {
 7:   if (argc > 1000000) {
 8:     int (* volatile tp)(void)=(int (*)(void))&t;
 9:     printf("%d", (*tp)());
10:   }
11: 
12:   return !!argv[argc];
13: }
14: 
15: int t(void) { ; return 0; }
/* end */

--------------------

• ruby -ryaml -e 'puts YAML.safe_load("---\nhello: world\n").inspect'
  {"hello"=>"world"}
• ruby -ryaml -e "puts Psych::Parser.new.respond_to?(:parse)"
  true
• ruby -ryaml -e 'YAML.safe_load("--- !ruby/object:Libv8::Node::Location::Vendor {}")'

/home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/psych-5.1.0/lib/psych/class_loader.rb:99:in `find': Tried to load unspecified class: Libv8::Node::Location::Vendor (Psych::DisallowedClass)
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/psych-5.1.0/lib/psych/class_loader.rb:28:in `load'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/psych-5.1.0/lib/psych/visitors/to_ruby.rb:426:in `resolve_class'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/psych-5.1.0/lib/psych/visitors/to_ruby.rb:215:in `visit_Psych_Nodes_Mapping'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/psych-5.1.0/lib/psych/visitors/visitor.rb:30:in `visit'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/psych-5.1.0/lib/psych/visitors/visitor.rb:6:in `accept'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/psych-5.1.0/lib/psych/visitors/to_ruby.rb:35:in `accept'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/psych-5.1.0/lib/psych/visitors/to_ruby.rb:320:in `visit_Psych_Nodes_Document'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/psych-5.1.0/lib/psych/visitors/visitor.rb:30:in `visit'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/psych-5.1.0/lib/psych/visitors/visitor.rb:6:in `accept'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/psych-5.1.0/lib/psych/visitors/to_ruby.rb:35:in `accept'
	from /home/ui/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/psych-5.1.0/lib/psych.rb:334:in `safe_load'
	from -e:1:in `<main>'

[flavorjones]

Closing the loop on this, a bug in library path resolution was reported at rubygems/rubygems#6611, fixed by rubygems/rubygems#6490, and the fix was released in rubygems 2.4.9.

TL;DR affected users can upgrade their rubygems installation to 2.4.9 or later.

I think we can close this discussion!

[dorianmariecom]

Thanks a lot @flavorjones