From b19881accfd9a138a2fdea620a96356efed226ab Mon Sep 17 00:00:00 2001 From: MaksimPW Date: Tue, 25 Apr 2023 17:29:29 +0700 Subject: [PATCH 01/13] init db on docker --- config/database.yml | 4 ++++ docker-compose.yml | 11 +++++++++++ 2 files changed, 15 insertions(+) create mode 100644 docker-compose.yml diff --git a/config/database.yml b/config/database.yml index e116cfa6..c5a00f15 100644 --- a/config/database.yml +++ b/config/database.yml @@ -16,7 +16,11 @@ # default: &default adapter: postgresql + host: localhost encoding: unicode + port: 5434 + username: postgres + password: postgres # For details on connection pooling, see Rails configuration guide # http://guides.rubyonrails.org/configuring.html#database-pooling pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %> diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 00000000..9b247aff --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,11 @@ +version: '3.9' +services: + postgres: + container_name: "optimization_task_3_postgres" + hostname: 'postgres' + image: 'postgres:14' + environment: + POSTGRES_USER: 'postgres' + POSTGRES_PASSWORD: 'postgres' + ports: + - "5434:5432" \ No newline at end of file From afe1f3c825090adbd5ad5bfd499fdc3d4f8b88fb Mon Sep 17 00:00:00 2001 From: MaksimPW Date: Tue, 25 Apr 2023 17:56:44 +0700 Subject: [PATCH 02/13] prepare work --- Makefile | 8 +++++++ case-study.md | 55 ++++++++++++++++++++++++++++++++++++++++++++ lib/tasks/utils.rake | 5 ++++ 3 files changed, 68 insertions(+) create mode 100644 Makefile create mode 100644 case-study.md diff --git a/Makefile b/Makefile new file mode 100644 index 00000000..dee83898 --- /dev/null +++ b/Makefile @@ -0,0 +1,8 @@ +reload_example: + bundle exec rake reload_json[fixtures/example.json] +reload_large: + bundle exec rake reload_json[fixtures/large.json] +reload_medium: + bundle exec rake reload_json[fixtures/medium.json] +reload_small: + bundle exec rake reload_json[fixtures/small.json] \ No newline at end of file diff --git a/case-study.md b/case-study.md new file mode 100644 index 00000000..7c3b94d2 --- /dev/null +++ b/case-study.md @@ -0,0 +1,55 @@ +# Case-study оптимизации + +## Актуальная проблема + +В нашем проекте возникла серьёзная проблема. + +Страница расписаний формируется не эффективно, механизм перезагрузки расписаний из файла занимает очень много времени(больше минуты) + +Нужно исправить это и сделать механизм перезагрузки расписаний быстрее, чтобы его обработка занимала меньше одной минуты + +## Формирование метрики +Для того, чтобы понимать, дают ли мои изменения положительный эффект на быстродействие программы я придумал использовать такую метрику: время исполнения rake таски и страницы с рейсами + +## Feedback-Loop +Для того, чтобы иметь возможность быстро проверять гипотезы я выстроил эффективный `feedback-loop`, который позволил мне получать обратную связь по эффективности сделанных изменений за N: + +- пока не выстроил + +## Вникаем в детали системы, чтобы найти главные точки роста +Для того, чтобы найти "точки роста" для оптимизации я воспользовался: + +- пока ничем не воспользовался + +Вот какие проблемы удалось найти и решить + +### Ваша находка №0 + +В работе + +## Результаты + +### rake reload_json + +| |ДО |ПОСЛЕ | +|------------------|--------------------------|--------------------------| +| example.json | 0.547757 | - | +| large.json | очень долго | - | +| medium.json | 229 сек | - | +| small.json | 30 сек | - | + +### Заметки + +#### Совет: как посчитать кол-во строк в файле +``` +wc -l data_large.rb # (3250940) total line count +``` + +#### Совет: как создать меньший файл из большего, оставив перевые N строк +``` +head -n N data_large.txt > dataN.txt +``` + +## Защита от регрессии производительности + +Пока ничего не сделано \ No newline at end of file diff --git a/lib/tasks/utils.rake b/lib/tasks/utils.rake index 540fe871..92f38043 100644 --- a/lib/tasks/utils.rake +++ b/lib/tasks/utils.rake @@ -1,6 +1,7 @@ # Наивная загрузка данных из json-файла в БД # rake reload_json[fixtures/small.json] task :reload_json, [:file_name] => :environment do |_task, args| + start_time = Time.now json = JSON.parse(File.read(args.file_name)) ActiveRecord::Base.transaction do @@ -29,6 +30,10 @@ task :reload_json, [:file_name] => :environment do |_task, args| duration_minutes: trip['duration_minutes'], price_cents: trip['price_cents'], ) + + print '.' end + + puts "\nFinished! Time: #{Time.now - start_time}" end end From 770ba092dd18474a24cc54b093afe7ba292cb357 Mon Sep 17 00:00:00 2001 From: MaksimPW Date: Tue, 25 Apr 2023 18:23:21 +0700 Subject: [PATCH 03/13] step 0: use activerecord-import for import trips --- Gemfile | 1 + Gemfile.lock | 3 +++ case-study.md | 14 +++++++++++++- lib/tasks/utils.rake | 8 +++++++- 4 files changed, 24 insertions(+), 2 deletions(-) diff --git a/Gemfile b/Gemfile index e20b1260..29bf9ea3 100644 --- a/Gemfile +++ b/Gemfile @@ -7,6 +7,7 @@ gem 'rails', '~> 5.2.3' gem 'pg', '>= 0.18', '< 2.0' gem 'puma', '~> 3.11' gem 'bootsnap', '>= 1.1.0', require: false +gem 'activerecord-import', require: false group :development, :test do # Call 'byebug' anywhere in the code to stop execution and get a debugger console diff --git a/Gemfile.lock b/Gemfile.lock index fccf6f5f..92bb058e 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -33,6 +33,8 @@ GEM activemodel (= 5.2.3) activesupport (= 5.2.3) arel (>= 9.0) + activerecord-import (1.4.1) + activerecord (>= 4.2) activestorage (5.2.3) actionpack (= 5.2.3) activerecord (= 5.2.3) @@ -134,6 +136,7 @@ PLATFORMS ruby DEPENDENCIES + activerecord-import bootsnap (>= 1.1.0) byebug listen (>= 3.0.5, < 3.2) diff --git a/case-study.md b/case-study.md index 7c3b94d2..69464ffa 100644 --- a/case-study.md +++ b/case-study.md @@ -25,7 +25,19 @@ ### Ваша находка №0 -В работе +Прогнал rake reload_json для всех файлов в fixtures, записал результаты +large.json отрабатывал слишком долго, не стал ждать +Буду ориентироваться на small.json, результат: 30 секунд + +Решил сразу воспользоваться советом использовать гем `activerecord-import` + +Поставил его, применил для импорта Trip +После прогона make reload_small время работы сократилось и составило 26 сек + +### Ваша находка №1 + + + ## Результаты diff --git a/lib/tasks/utils.rake b/lib/tasks/utils.rake index 92f38043..0df05d43 100644 --- a/lib/tasks/utils.rake +++ b/lib/tasks/utils.rake @@ -1,5 +1,7 @@ # Наивная загрузка данных из json-файла в БД # rake reload_json[fixtures/small.json] +require 'activerecord-import' + task :reload_json, [:file_name] => :environment do |_task, args| start_time = Time.now json = JSON.parse(File.read(args.file_name)) @@ -11,6 +13,8 @@ task :reload_json, [:file_name] => :environment do |_task, args| Trip.delete_all ActiveRecord::Base.connection.execute('delete from buses_services;') + trips = [] + json.each do |trip| from = City.find_or_create_by(name: trip['from']) to = City.find_or_create_by(name: trip['to']) @@ -22,7 +26,7 @@ task :reload_json, [:file_name] => :environment do |_task, args| bus = Bus.find_or_create_by(number: trip['bus']['number']) bus.update(model: trip['bus']['model'], services: services) - Trip.create!( + trips << Trip.new( from: from, to: to, bus: bus, @@ -34,6 +38,8 @@ task :reload_json, [:file_name] => :environment do |_task, args| print '.' end + Trip.import trips + puts "\nFinished! Time: #{Time.now - start_time}" end end From f6bb2089b294243b6d336537fd30f06ec9ede9fa Mon Sep 17 00:00:00 2001 From: MaksimPW Date: Tue, 25 Apr 2023 18:55:20 +0700 Subject: [PATCH 04/13] init pghero --- Gemfile | 1 + Gemfile.lock | 3 +++ Makefile | 5 ++++- config/routes.rb | 1 + docker-compose.yml | 3 ++- 5 files changed, 11 insertions(+), 2 deletions(-) diff --git a/Gemfile b/Gemfile index 29bf9ea3..c9c6433f 100644 --- a/Gemfile +++ b/Gemfile @@ -8,6 +8,7 @@ gem 'pg', '>= 0.18', '< 2.0' gem 'puma', '~> 3.11' gem 'bootsnap', '>= 1.1.0', require: false gem 'activerecord-import', require: false +gem 'pghero' group :development, :test do # Call 'byebug' anywhere in the code to stop execution and get a debugger console diff --git a/Gemfile.lock b/Gemfile.lock index 92bb058e..1494d7a3 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -79,6 +79,8 @@ GEM nokogiri (1.10.2) mini_portile2 (~> 2.4.0) pg (1.1.4) + pghero (2.8.3) + activerecord (>= 5) puma (3.12.1) rack (2.0.6) rack-test (1.1.0) @@ -141,6 +143,7 @@ DEPENDENCIES byebug listen (>= 3.0.5, < 3.2) pg (>= 0.18, < 2.0) + pghero puma (~> 3.11) rails (~> 5.2.3) tzinfo-data diff --git a/Makefile b/Makefile index dee83898..a00a62aa 100644 --- a/Makefile +++ b/Makefile @@ -5,4 +5,7 @@ reload_large: reload_medium: bundle exec rake reload_json[fixtures/medium.json] reload_small: - bundle exec rake reload_json[fixtures/small.json] \ No newline at end of file + bundle exec rake reload_json[fixtures/small.json] + +pghero: + open http://localhost:3000/pghero \ No newline at end of file diff --git a/config/routes.rb b/config/routes.rb index a2da6a7b..7f620876 100644 --- a/config/routes.rb +++ b/config/routes.rb @@ -2,4 +2,5 @@ # For details on the DSL available within this file, see http://guides.rubyonrails.org/routing.html get "/" => "statistics#index" get "автобусы/:from/:to" => "trips#index" + mount PgHero::Engine, at: "pghero" end diff --git a/docker-compose.yml b/docker-compose.yml index 9b247aff..e28f7b93 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -8,4 +8,5 @@ services: POSTGRES_USER: 'postgres' POSTGRES_PASSWORD: 'postgres' ports: - - "5434:5432" \ No newline at end of file + - "5434:5432" + command: postgres -c shared_preload_libraries='pg_stat_statements' -c pg_stat_statements.track=all From 0f41284be9eb959d499f1bad721d9bbb4167ad3c Mon Sep 17 00:00:00 2001 From: MaksimPW Date: Tue, 25 Apr 2023 19:06:49 +0700 Subject: [PATCH 05/13] init rack-mini-profiler --- Gemfile | 5 +++++ Gemfile.lock | 7 +++++++ Makefile | 2 ++ 3 files changed, 14 insertions(+) diff --git a/Gemfile b/Gemfile index c9c6433f..27a6c788 100644 --- a/Gemfile +++ b/Gemfile @@ -8,7 +8,12 @@ gem 'pg', '>= 0.18', '< 2.0' gem 'puma', '~> 3.11' gem 'bootsnap', '>= 1.1.0', require: false gem 'activerecord-import', require: false + +# Profiling gem 'pghero' +gem 'rack-mini-profiler' +gem 'memory_profiler' +gem 'stackprof' group :development, :test do # Call 'byebug' anywhere in the code to stop execution and get a debugger console diff --git a/Gemfile.lock b/Gemfile.lock index 1494d7a3..1735cd8e 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -69,6 +69,7 @@ GEM mini_mime (>= 0.1.1) marcel (0.3.3) mimemagic (~> 0.3.2) + memory_profiler (1.0.1) method_source (0.9.2) mimemagic (0.3.3) mini_mime (1.0.1) @@ -83,6 +84,8 @@ GEM activerecord (>= 5) puma (3.12.1) rack (2.0.6) + rack-mini-profiler (3.1.0) + rack (>= 1.2.0) rack-test (1.1.0) rack (>= 1.0, < 3) rails (5.2.3) @@ -121,6 +124,7 @@ GEM actionpack (>= 4.0) activesupport (>= 4.0) sprockets (>= 3.0.0) + stackprof (0.2.25) thor (0.20.3) thread_safe (0.3.6) tzinfo (1.2.5) @@ -142,10 +146,13 @@ DEPENDENCIES bootsnap (>= 1.1.0) byebug listen (>= 3.0.5, < 3.2) + memory_profiler pg (>= 0.18, < 2.0) pghero puma (~> 3.11) + rack-mini-profiler rails (~> 5.2.3) + stackprof tzinfo-data web-console (>= 3.3.0) diff --git a/Makefile b/Makefile index a00a62aa..6bd62239 100644 --- a/Makefile +++ b/Makefile @@ -7,5 +7,7 @@ reload_medium: reload_small: bundle exec rake reload_json[fixtures/small.json] +open: + open http://localhost:3000/автобусы/Самара/Москва pghero: open http://localhost:3000/pghero \ No newline at end of file From 1e2047dd6336a00295b500b2f9ceb3e93c51d0b6 Mon Sep 17 00:00:00 2001 From: MaksimPW Date: Wed, 26 Apr 2023 01:18:47 +0700 Subject: [PATCH 06/13] step 1: prepare optimization --- .gitignore | 4 + .rspec | 1 + Gemfile | 7 +- Gemfile.lock | 39 ++++++++++ Makefile | 8 +- app/interactions/import_data.rb | 63 +++++++++++++++ bin/bundle | 106 +++++++++++++++++++++++++- bin/byebug | 29 +++++++ bin/htmldiff | 29 +++++++ bin/ldiff | 29 +++++++ bin/listen | 29 +++++++ bin/nokogiri | 29 +++++++ bin/puma | 29 +++++++ bin/pumactl | 29 +++++++ bin/rackup | 29 +++++++ bin/rails | 34 +++++++-- bin/rake | 34 +++++++-- bin/rspec | 29 +++++++ bin/ruby-memory-profiler | 29 +++++++ bin/sprockets | 29 +++++++ bin/stackprof | 29 +++++++ bin/stackprof-flamegraph.pl | 29 +++++++ bin/stackprof-gprof2dot.py | 29 +++++++ bin/thor | 29 +++++++ case-study.md | 5 +- lib/tasks/utils.rake | 47 +++--------- spec/interactions/import_data_spec.rb | 17 +++++ spec/rails_helper.rb | 64 ++++++++++++++++ spec/spec_helper.rb | 94 +++++++++++++++++++++++ 29 files changed, 900 insertions(+), 58 deletions(-) create mode 100644 .rspec create mode 100644 app/interactions/import_data.rb create mode 100755 bin/byebug create mode 100755 bin/htmldiff create mode 100755 bin/ldiff create mode 100755 bin/listen create mode 100755 bin/nokogiri create mode 100755 bin/puma create mode 100755 bin/pumactl create mode 100755 bin/rackup create mode 100755 bin/rspec create mode 100755 bin/ruby-memory-profiler create mode 100755 bin/sprockets create mode 100755 bin/stackprof create mode 100755 bin/stackprof-flamegraph.pl create mode 100755 bin/stackprof-gprof2dot.py create mode 100755 bin/thor create mode 100644 spec/interactions/import_data_spec.rb create mode 100644 spec/rails_helper.rb create mode 100644 spec/spec_helper.rb diff --git a/.gitignore b/.gitignore index 59c74047..6c3edac2 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,7 @@ /tmp /log /public + +.byebug_history + +reports/tmp/* \ No newline at end of file diff --git a/.rspec b/.rspec new file mode 100644 index 00000000..c99d2e73 --- /dev/null +++ b/.rspec @@ -0,0 +1 @@ +--require spec_helper diff --git a/Gemfile b/Gemfile index 27a6c788..217e84f6 100644 --- a/Gemfile +++ b/Gemfile @@ -7,7 +7,9 @@ gem 'rails', '~> 5.2.3' gem 'pg', '>= 0.18', '< 2.0' gem 'puma', '~> 3.11' gem 'bootsnap', '>= 1.1.0', require: false -gem 'activerecord-import', require: false +gem 'activerecord-import' +gem 'oj', require: false +gem 'active_interaction' # Profiling gem 'pghero' @@ -27,6 +29,9 @@ group :development do end group :test do + gem 'rspec' + gem 'rspec-rails' + gem 'rspec-benchmark' end # Windows does not include zoneinfo files, so bundle the tzinfo-data gem diff --git a/Gemfile.lock b/Gemfile.lock index 1735cd8e..e81ed295 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -24,6 +24,9 @@ GEM erubi (~> 1.4) rails-dom-testing (~> 2.0) rails-html-sanitizer (~> 1.0, >= 1.0.3) + active_interaction (4.1.0) + activemodel (>= 5, < 8) + activesupport (>= 5, < 8) activejob (5.2.3) activesupport (= 5.2.3) globalid (>= 0.3.6) @@ -45,6 +48,9 @@ GEM minitest (~> 5.1) tzinfo (~> 1.1) arel (9.0.0) + benchmark-malloc (0.2.0) + benchmark-perf (0.6.0) + benchmark-trend (0.4.0) bindex (0.6.0) bootsnap (1.4.2) msgpack (~> 1.0) @@ -52,6 +58,7 @@ GEM byebug (11.0.1) concurrent-ruby (1.1.5) crass (1.0.4) + diff-lcs (1.5.0) erubi (1.8.0) ffi (1.10.0) globalid (0.4.2) @@ -79,6 +86,7 @@ GEM nio4r (2.3.1) nokogiri (1.10.2) mini_portile2 (~> 2.4.0) + oj (3.14.2) pg (1.1.4) pghero (2.8.3) activerecord (>= 5) @@ -116,6 +124,32 @@ GEM rb-fsevent (0.10.3) rb-inotify (0.10.0) ffi (~> 1.0) + rspec (3.12.0) + rspec-core (~> 3.12.0) + rspec-expectations (~> 3.12.0) + rspec-mocks (~> 3.12.0) + rspec-benchmark (0.6.0) + benchmark-malloc (~> 0.2) + benchmark-perf (~> 0.6) + benchmark-trend (~> 0.4) + rspec (>= 3.0) + rspec-core (3.12.2) + rspec-support (~> 3.12.0) + rspec-expectations (3.12.3) + diff-lcs (>= 1.2.0, < 2.0) + rspec-support (~> 3.12.0) + rspec-mocks (3.12.5) + diff-lcs (>= 1.2.0, < 2.0) + rspec-support (~> 3.12.0) + rspec-rails (5.1.2) + actionpack (>= 5.2) + activesupport (>= 5.2) + railties (>= 5.2) + rspec-core (~> 3.10) + rspec-expectations (~> 3.10) + rspec-mocks (~> 3.10) + rspec-support (~> 3.10) + rspec-support (3.12.0) ruby_dep (1.5.0) sprockets (3.7.2) concurrent-ruby (~> 1.0) @@ -142,16 +176,21 @@ PLATFORMS ruby DEPENDENCIES + active_interaction activerecord-import bootsnap (>= 1.1.0) byebug listen (>= 3.0.5, < 3.2) memory_profiler + oj pg (>= 0.18, < 2.0) pghero puma (~> 3.11) rack-mini-profiler rails (~> 5.2.3) + rspec + rspec-benchmark + rspec-rails stackprof tzinfo-data web-console (>= 3.3.0) diff --git a/Makefile b/Makefile index 6bd62239..f5090aba 100644 --- a/Makefile +++ b/Makefile @@ -10,4 +10,10 @@ reload_small: open: open http://localhost:3000/автобусы/Самара/Москва pghero: - open http://localhost:3000/pghero \ No newline at end of file + open http://localhost:3000/pghero + +profile_memory: + bundle exec rake reload_json[fixtures/example.json,memory] + +mv_reports: + mv reports/tmp reports/${step} \ No newline at end of file diff --git a/app/interactions/import_data.rb b/app/interactions/import_data.rb new file mode 100644 index 00000000..9af74daf --- /dev/null +++ b/app/interactions/import_data.rb @@ -0,0 +1,63 @@ +# frozen_string_literal: true +require 'activerecord-import' + +class ImportData < ActiveInteraction::Base + string :file_name + + def execute + start_time = Time.now + + ActiveRecord::Base.transaction do + drop_data + import_data + end + + puts "\nFinished! Time: #{Time.now - start_time}" + print_memory_usage + end + + private + + def drop_data + City.delete_all + Bus.delete_all + Service.delete_all + Trip.delete_all + ActiveRecord::Base.connection.execute('delete from buses_services;') + end + + def import_data + json = JSON.parse(File.read(file_name)) + + trips = [] + + json.each do |trip| + from = City.find_or_create_by(name: trip['from']) + to = City.find_or_create_by(name: trip['to']) + services = [] + trip['bus']['services'].each do |service| + s = Service.find_or_create_by(name: service) + services << s + end + bus = Bus.find_or_create_by(number: trip['bus']['number']) + bus.update(model: trip['bus']['model'], services: services) + + trips << Trip.new( + from: from, + to: to, + bus: bus, + start_time: trip['start_time'], + duration_minutes: trip['duration_minutes'], + price_cents: trip['price_cents'], + ) + + print '.' + end + + Trip.import trips + end + + def print_memory_usage + puts "MEMORY USAGE: %d MB" % (`ps -o rss= -p #{Process.pid}`.to_i / 1024) + end +end \ No newline at end of file diff --git a/bin/bundle b/bin/bundle index f19acf5b..524dfd3f 100755 --- a/bin/bundle +++ b/bin/bundle @@ -1,3 +1,105 @@ #!/usr/bin/env ruby -ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../Gemfile', __dir__) -load Gem.bin_path('bundler', 'bundle') +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'bundle' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "rubygems" + +m = Module.new do + module_function + + def invoked_as_script? + File.expand_path($0) == File.expand_path(__FILE__) + end + + def env_var_version + ENV["BUNDLER_VERSION"] + end + + def cli_arg_version + return unless invoked_as_script? # don't want to hijack other binstubs + return unless "update".start_with?(ARGV.first || " ") # must be running `bundle update` + bundler_version = nil + update_index = nil + ARGV.each_with_index do |a, i| + if update_index && update_index.succ == i && a =~ Gem::Version::ANCHORED_VERSION_PATTERN + bundler_version = a + end + next unless a =~ /\A--bundler(?:[= ](#{Gem::Version::VERSION_PATTERN}))?\z/ + bundler_version = $1 || ">= 0.a" + update_index = i + end + bundler_version + end + + def gemfile + gemfile = ENV["BUNDLE_GEMFILE"] + return gemfile if gemfile && !gemfile.empty? + + File.expand_path("../../Gemfile", __FILE__) + end + + def lockfile + lockfile = + case File.basename(gemfile) + when "gems.rb" then gemfile.sub(/\.rb$/, gemfile) + else "#{gemfile}.lock" + end + File.expand_path(lockfile) + end + + def lockfile_version + return unless File.file?(lockfile) + lockfile_contents = File.read(lockfile) + return unless lockfile_contents =~ /\n\nBUNDLED WITH\n\s{2,}(#{Gem::Version::VERSION_PATTERN})\n/ + Regexp.last_match(1) + end + + def bundler_version + @bundler_version ||= begin + env_var_version || cli_arg_version || + lockfile_version || "#{Gem::Requirement.default}.a" + end + end + + def load_bundler! + ENV["BUNDLE_GEMFILE"] ||= gemfile + + # must dup string for RG < 1.8 compatibility + activate_bundler(bundler_version.dup) + end + + def activate_bundler(bundler_version) + if Gem::Version.correct?(bundler_version) && Gem::Version.new(bundler_version).release < Gem::Version.new("2.0") + bundler_version = "< 2" + end + gem_error = activation_error_handling do + gem "bundler", bundler_version + end + return if gem_error.nil? + require_error = activation_error_handling do + require "bundler/version" + end + return if require_error.nil? && Gem::Requirement.new(bundler_version).satisfied_by?(Gem::Version.new(Bundler::VERSION)) + warn "Activating bundler (#{bundler_version}) failed:\n#{gem_error.message}\n\nTo install the version of bundler this project requires, run `gem install bundler -v '#{bundler_version}'`" + exit 42 + end + + def activation_error_handling + yield + nil + rescue StandardError, LoadError => e + e + end +end + +m.load_bundler! + +if m.invoked_as_script? + load Gem.bin_path("bundler", "bundle") +end diff --git a/bin/byebug b/bin/byebug new file mode 100755 index 00000000..16f20bbd --- /dev/null +++ b/bin/byebug @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'byebug' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("byebug", "byebug") diff --git a/bin/htmldiff b/bin/htmldiff new file mode 100755 index 00000000..091820c9 --- /dev/null +++ b/bin/htmldiff @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'htmldiff' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("diff-lcs", "htmldiff") diff --git a/bin/ldiff b/bin/ldiff new file mode 100755 index 00000000..073e19f2 --- /dev/null +++ b/bin/ldiff @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'ldiff' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("diff-lcs", "ldiff") diff --git a/bin/listen b/bin/listen new file mode 100755 index 00000000..3dd36a7e --- /dev/null +++ b/bin/listen @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'listen' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("listen", "listen") diff --git a/bin/nokogiri b/bin/nokogiri new file mode 100755 index 00000000..b22a1a0a --- /dev/null +++ b/bin/nokogiri @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'nokogiri' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("nokogiri", "nokogiri") diff --git a/bin/puma b/bin/puma new file mode 100755 index 00000000..880935b2 --- /dev/null +++ b/bin/puma @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'puma' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("puma", "puma") diff --git a/bin/pumactl b/bin/pumactl new file mode 100755 index 00000000..b698e6e9 --- /dev/null +++ b/bin/pumactl @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'pumactl' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("puma", "pumactl") diff --git a/bin/rackup b/bin/rackup new file mode 100755 index 00000000..3ac4a5a7 --- /dev/null +++ b/bin/rackup @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'rackup' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("rack", "rackup") diff --git a/bin/rails b/bin/rails index 5badb2fd..7fd59cc7 100755 --- a/bin/rails +++ b/bin/rails @@ -1,9 +1,29 @@ #!/usr/bin/env ruby -begin - load File.expand_path('../spring', __FILE__) -rescue LoadError => e - raise unless e.message.include?('spring') +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'rails' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end end -APP_PATH = File.expand_path('../config/application', __dir__) -require_relative '../config/boot' -require 'rails/commands' + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("railties", "rails") diff --git a/bin/rake b/bin/rake index d87d5f57..9275675e 100755 --- a/bin/rake +++ b/bin/rake @@ -1,9 +1,29 @@ #!/usr/bin/env ruby -begin - load File.expand_path('../spring', __FILE__) -rescue LoadError => e - raise unless e.message.include?('spring') +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'rake' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end end -require_relative '../config/boot' -require 'rake' -Rake.application.run + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("rake", "rake") diff --git a/bin/rspec b/bin/rspec new file mode 100755 index 00000000..a6c78521 --- /dev/null +++ b/bin/rspec @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'rspec' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("rspec-core", "rspec") diff --git a/bin/ruby-memory-profiler b/bin/ruby-memory-profiler new file mode 100755 index 00000000..b13a4b03 --- /dev/null +++ b/bin/ruby-memory-profiler @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'ruby-memory-profiler' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("memory_profiler", "ruby-memory-profiler") diff --git a/bin/sprockets b/bin/sprockets new file mode 100755 index 00000000..9f75aa74 --- /dev/null +++ b/bin/sprockets @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'sprockets' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("sprockets", "sprockets") diff --git a/bin/stackprof b/bin/stackprof new file mode 100755 index 00000000..cfc10350 --- /dev/null +++ b/bin/stackprof @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'stackprof' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("stackprof", "stackprof") diff --git a/bin/stackprof-flamegraph.pl b/bin/stackprof-flamegraph.pl new file mode 100755 index 00000000..f6212797 --- /dev/null +++ b/bin/stackprof-flamegraph.pl @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'stackprof-flamegraph.pl' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("stackprof", "stackprof-flamegraph.pl") diff --git a/bin/stackprof-gprof2dot.py b/bin/stackprof-gprof2dot.py new file mode 100755 index 00000000..e72dd002 --- /dev/null +++ b/bin/stackprof-gprof2dot.py @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'stackprof-gprof2dot.py' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("stackprof", "stackprof-gprof2dot.py") diff --git a/bin/thor b/bin/thor new file mode 100755 index 00000000..71bfaeae --- /dev/null +++ b/bin/thor @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'thor' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("thor", "thor") diff --git a/case-study.md b/case-study.md index 69464ffa..11c8fb97 100644 --- a/case-study.md +++ b/case-study.md @@ -36,8 +36,9 @@ large.json отрабатывал слишком долго, не стал жд ### Ваша находка №1 - - +Добавил в проект pghero, rack-mini-profiler, memory-profiler +Перенес код по импорту данных из rake task в отдельный интерактор ImportData +Написал rspec тесты ## Результаты diff --git a/lib/tasks/utils.rake b/lib/tasks/utils.rake index 0df05d43..d2d8d500 100644 --- a/lib/tasks/utils.rake +++ b/lib/tasks/utils.rake @@ -1,45 +1,16 @@ # Наивная загрузка данных из json-файла в БД # rake reload_json[fixtures/small.json] -require 'activerecord-import' -task :reload_json, [:file_name] => :environment do |_task, args| - start_time = Time.now - json = JSON.parse(File.read(args.file_name)) - - ActiveRecord::Base.transaction do - City.delete_all - Bus.delete_all - Service.delete_all - Trip.delete_all - ActiveRecord::Base.connection.execute('delete from buses_services;') - - trips = [] - - json.each do |trip| - from = City.find_or_create_by(name: trip['from']) - to = City.find_or_create_by(name: trip['to']) - services = [] - trip['bus']['services'].each do |service| - s = Service.find_or_create_by(name: service) - services << s - end - bus = Bus.find_or_create_by(number: trip['bus']['number']) - bus.update(model: trip['bus']['model'], services: services) - - trips << Trip.new( - from: from, - to: to, - bus: bus, - start_time: trip['start_time'], - duration_minutes: trip['duration_minutes'], - price_cents: trip['price_cents'], - ) - - print '.' +task :reload_json, [:file_name, :profiler] => :environment do |_task, args| + case args.profiler + when 'memory' + report = MemoryProfiler.report do + ImportData.run!(file_name: args.file_name) end - Trip.import trips - - puts "\nFinished! Time: #{Time.now - start_time}" + report.pretty_print(scale_bytes: true, color_output: true) + report.pretty_print(scale_bytes: true, to_file: 'reports/tmp/memory_profiler.txt') + else + ImportData.run!(file_name: args.file_name) end end diff --git a/spec/interactions/import_data_spec.rb b/spec/interactions/import_data_spec.rb new file mode 100644 index 00000000..670e7d46 --- /dev/null +++ b/spec/interactions/import_data_spec.rb @@ -0,0 +1,17 @@ +require 'rails_helper' + +RSpec.describe ImportData do + subject { described_class.run!(file_name: 'fixtures/example.json') } + + describe '#execute' do + context 'common' do + it 'creates trips' do + expect { subject } + .to change { Trip.count }.by(10) + .and change { City.count }.by(2) + .and change { Service.count }.by(2) + .and change { Bus.count }.by(1) + end + end + end +end diff --git a/spec/rails_helper.rb b/spec/rails_helper.rb new file mode 100644 index 00000000..b6317b5a --- /dev/null +++ b/spec/rails_helper.rb @@ -0,0 +1,64 @@ +# This file is copied to spec/ when you run 'rails generate rspec:install' +require 'spec_helper' +ENV['RAILS_ENV'] ||= 'test' +require_relative '../config/environment' +# Prevent database truncation if the environment is production +abort("The Rails environment is running in production mode!") if Rails.env.production? +require 'rspec/rails' +# Add additional requires below this line. Rails is not loaded until this point! + +# Requires supporting ruby files with custom matchers and macros, etc, in +# spec/support/ and its subdirectories. Files matching `spec/**/*_spec.rb` are +# run as spec files by default. This means that files in spec/support that end +# in _spec.rb will both be required and run as specs, causing the specs to be +# run twice. It is recommended that you do not name files matching this glob to +# end with _spec.rb. You can configure this pattern with the --pattern +# option on the command line or in ~/.rspec, .rspec or `.rspec-local`. +# +# The following line is provided for convenience purposes. It has the downside +# of increasing the boot-up time by auto-requiring all files in the support +# directory. Alternatively, in the individual `*_spec.rb` files, manually +# require only the support files necessary. +# +# Dir[Rails.root.join('spec', 'support', '**', '*.rb')].sort.each { |f| require f } + +# Checks for pending migrations and applies them before tests are run. +# If you are not using ActiveRecord, you can remove these lines. +begin + ActiveRecord::Migration.maintain_test_schema! +rescue ActiveRecord::PendingMigrationError => e + puts e.to_s.strip + exit 1 +end +RSpec.configure do |config| + # Remove this line if you're not using ActiveRecord or ActiveRecord fixtures + config.fixture_path = "#{::Rails.root}/spec/fixtures" + + # If you're not using ActiveRecord, or you'd prefer not to run each of your + # examples within a transaction, remove the following line or assign false + # instead of true. + config.use_transactional_fixtures = true + + # You can uncomment this line to turn off ActiveRecord support entirely. + # config.use_active_record = false + + # RSpec Rails can automatically mix in different behaviours to your tests + # based on their file location, for example enabling you to call `get` and + # `post` in specs under `spec/controllers`. + # + # You can disable this behaviour by removing the line below, and instead + # explicitly tag your specs with their type, e.g.: + # + # RSpec.describe UsersController, type: :controller do + # # ... + # end + # + # The different available types are documented in the features, such as in + # https://relishapp.com/rspec/rspec-rails/docs + config.infer_spec_type_from_file_location! + + # Filter lines from Rails gems in backtraces. + config.filter_rails_from_backtrace! + # arbitrary gems may also be filtered via: + # config.filter_gems_from_backtrace("gem name") +end diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb new file mode 100644 index 00000000..327b58ea --- /dev/null +++ b/spec/spec_helper.rb @@ -0,0 +1,94 @@ +# This file was generated by the `rails generate rspec:install` command. Conventionally, all +# specs live under a `spec` directory, which RSpec adds to the `$LOAD_PATH`. +# The generated `.rspec` file contains `--require spec_helper` which will cause +# this file to always be loaded, without a need to explicitly require it in any +# files. +# +# Given that it is always loaded, you are encouraged to keep this file as +# light-weight as possible. Requiring heavyweight dependencies from this file +# will add to the boot time of your test suite on EVERY test run, even for an +# individual file that may not need all of that loaded. Instead, consider making +# a separate helper file that requires the additional dependencies and performs +# the additional setup, and require it from the spec files that actually need +# it. +# +# See https://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration +RSpec.configure do |config| + # rspec-expectations config goes here. You can use an alternate + # assertion/expectation library such as wrong or the stdlib/minitest + # assertions if you prefer. + config.expect_with :rspec do |expectations| + # This option will default to `true` in RSpec 4. It makes the `description` + # and `failure_message` of custom matchers include text for helper methods + # defined using `chain`, e.g.: + # be_bigger_than(2).and_smaller_than(4).description + # # => "be bigger than 2 and smaller than 4" + # ...rather than: + # # => "be bigger than 2" + expectations.include_chain_clauses_in_custom_matcher_descriptions = true + end + + # rspec-mocks config goes here. You can use an alternate test double + # library (such as bogus or mocha) by changing the `mock_with` option here. + config.mock_with :rspec do |mocks| + # Prevents you from mocking or stubbing a method that does not exist on + # a real object. This is generally recommended, and will default to + # `true` in RSpec 4. + mocks.verify_partial_doubles = true + end + + # This option will default to `:apply_to_host_groups` in RSpec 4 (and will + # have no way to turn it off -- the option exists only for backwards + # compatibility in RSpec 3). It causes shared context metadata to be + # inherited by the metadata hash of host groups and examples, rather than + # triggering implicit auto-inclusion in groups with matching metadata. + config.shared_context_metadata_behavior = :apply_to_host_groups + +# The settings below are suggested to provide a good initial experience +# with RSpec, but feel free to customize to your heart's content. +=begin + # This allows you to limit a spec run to individual examples or groups + # you care about by tagging them with `:focus` metadata. When nothing + # is tagged with `:focus`, all examples get run. RSpec also provides + # aliases for `it`, `describe`, and `context` that include `:focus` + # metadata: `fit`, `fdescribe` and `fcontext`, respectively. + config.filter_run_when_matching :focus + + # Allows RSpec to persist some state between runs in order to support + # the `--only-failures` and `--next-failure` CLI options. We recommend + # you configure your source control system to ignore this file. + config.example_status_persistence_file_path = "spec/examples.txt" + + # Limits the available syntax to the non-monkey patched syntax that is + # recommended. For more details, see: + # https://rspec.info/features/3-12/rspec-core/configuration/zero-monkey-patching-mode/ + config.disable_monkey_patching! + + # Many RSpec users commonly either run the entire suite or an individual + # file, and it's useful to allow more verbose output when running an + # individual spec file. + if config.files_to_run.one? + # Use the documentation formatter for detailed output, + # unless a formatter has already been configured + # (e.g. via a command-line flag). + config.default_formatter = "doc" + end + + # Print the 10 slowest examples and example groups at the + # end of the spec run, to help surface which specs are running + # particularly slow. + config.profile_examples = 10 + + # Run specs in random order to surface order dependencies. If you find an + # order dependency and want to debug it, you can fix the order by providing + # the seed, which is printed after each run. + # --seed 1234 + config.order = :random + + # Seed global randomization in this process using the `--seed` CLI option. + # Setting this allows you to use `--seed` to deterministically reproduce + # test failures related to randomization by passing the same `--seed` value + # as the one that triggered the failure. + Kernel.srand config.seed +=end +end From ab640be6b5dc661dcc7b7f10aef7fa28033fcccc Mon Sep 17 00:00:00 2001 From: MaksimPW Date: Wed, 26 Apr 2023 01:46:53 +0700 Subject: [PATCH 07/13] step 1: init bullet --- Gemfile | 1 + Gemfile.lock | 5 +++++ Readme.md | 1 + case-study.md | 2 +- config/environments/development.rb | 9 +++++++++ config/environments/test.rb | 6 ++++++ 6 files changed, 23 insertions(+), 1 deletion(-) diff --git a/Gemfile b/Gemfile index 217e84f6..884a0f5a 100644 --- a/Gemfile +++ b/Gemfile @@ -18,6 +18,7 @@ gem 'memory_profiler' gem 'stackprof' group :development, :test do + gem 'bullet' # Call 'byebug' anywhere in the code to stop execution and get a debugger console gem 'byebug', platforms: [:mri, :mingw, :x64_mingw] end diff --git a/Gemfile.lock b/Gemfile.lock index e81ed295..37709380 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -55,6 +55,9 @@ GEM bootsnap (1.4.2) msgpack (~> 1.0) builder (3.2.3) + bullet (7.0.7) + activesupport (>= 3.0.0) + uniform_notifier (~> 1.11) byebug (11.0.1) concurrent-ruby (1.1.5) crass (1.0.4) @@ -163,6 +166,7 @@ GEM thread_safe (0.3.6) tzinfo (1.2.5) thread_safe (~> 0.1) + uniform_notifier (1.16.0) web-console (3.7.0) actionview (>= 5.0) activemodel (>= 5.0) @@ -179,6 +183,7 @@ DEPENDENCIES active_interaction activerecord-import bootsnap (>= 1.1.0) + bullet byebug listen (>= 3.0.5, < 3.2) memory_profiler diff --git a/Readme.md b/Readme.md index 20b4eda3..faf4e797 100644 --- a/Readme.md +++ b/Readme.md @@ -47,6 +47,7 @@ - [ ] `rails panel` - [ ] `bullet` - [ ] `explain` запросов +- [ ] `rspec-sqlimit` ### Сдача задания `PR` в этот репозиторий с кодом и case-study наподобие первых двух недель. На этот раз шаблона нет, законспектируйте ваш процесс оптимизации в свободной форме. diff --git a/case-study.md b/case-study.md index 11c8fb97..7952a8bf 100644 --- a/case-study.md +++ b/case-study.md @@ -36,7 +36,7 @@ large.json отрабатывал слишком долго, не стал жд ### Ваша находка №1 -Добавил в проект pghero, rack-mini-profiler, memory-profiler +Добавил в проект pghero, rack-mini-profiler, memory-profiler, bullet Перенес код по импорту данных из rake task в отдельный интерактор ImportData Написал rspec тесты diff --git a/config/environments/development.rb b/config/environments/development.rb index 1311e3e4..903a8661 100644 --- a/config/environments/development.rb +++ b/config/environments/development.rb @@ -1,4 +1,13 @@ Rails.application.configure do + config.after_initialize do + Bullet.enable = true + Bullet.alert = true + Bullet.bullet_logger = true + Bullet.console = true + Bullet.rails_logger = true + Bullet.add_footer = true + end + # Settings specified here will take precedence over those in config/application.rb. # In the development environment your application's code is reloaded on diff --git a/config/environments/test.rb b/config/environments/test.rb index 0a38fd3c..bc5971ab 100644 --- a/config/environments/test.rb +++ b/config/environments/test.rb @@ -1,4 +1,10 @@ Rails.application.configure do + config.after_initialize do + Bullet.enable = true + Bullet.bullet_logger = true + Bullet.raise = true # raise an error if n+1 query occurs + end + # Settings specified here will take precedence over those in config/application.rb. # The test environment is used exclusively to run your application's From 0314a9379b8fe24d3dfd7fbd7cebe0e3e169b02e Mon Sep 17 00:00:00 2001 From: MaksimPW Date: Wed, 26 Apr 2023 02:29:11 +0700 Subject: [PATCH 08/13] step 1: refactoring --- app/interactions/import_data.rb | 47 ++++++++++++++++++++------------- case-study.md | 6 ++++- 2 files changed, 33 insertions(+), 20 deletions(-) diff --git a/app/interactions/import_data.rb b/app/interactions/import_data.rb index 9af74daf..84dd1915 100644 --- a/app/interactions/import_data.rb +++ b/app/interactions/import_data.rb @@ -32,31 +32,40 @@ def import_data trips = [] json.each do |trip| - from = City.find_or_create_by(name: trip['from']) - to = City.find_or_create_by(name: trip['to']) - services = [] - trip['bus']['services'].each do |service| - s = Service.find_or_create_by(name: service) - services << s - end - bus = Bus.find_or_create_by(number: trip['bus']['number']) - bus.update(model: trip['bus']['model'], services: services) - - trips << Trip.new( - from: from, - to: to, - bus: bus, - start_time: trip['start_time'], - duration_minutes: trip['duration_minutes'], - price_cents: trip['price_cents'], - ) - + trips << trip_new(trip) print '.' end Trip.import trips end + def trip_new(trip) + Trip.new( + from: fetch_city(trip['from']), + to: fetch_city(trip['to']), + bus: fetch_bus(trip), + start_time: trip['start_time'], + duration_minutes: trip['duration_minutes'], + price_cents: trip['price_cents'], + ) + end + + def fetch_city(name) + City.find_or_create_by(name: name) + end + + def fetch_bus(trip) + bus = Bus.find_or_create_by(number: trip['bus']['number']) + bus.update(model: trip['bus']['model'], services: fetch_bus_services(trip)) + bus + end + + def fetch_bus_services(trip) + trip['bus']['services'].map do |service| + Service.find_or_create_by(name: service) + end + end + def print_memory_usage puts "MEMORY USAGE: %d MB" % (`ps -o rss= -p #{Process.pid}`.to_i / 1024) end diff --git a/case-study.md b/case-study.md index 7952a8bf..61f62b88 100644 --- a/case-study.md +++ b/case-study.md @@ -36,9 +36,13 @@ large.json отрабатывал слишком долго, не стал жд ### Ваша находка №1 +Подготовка: + Добавил в проект pghero, rack-mini-profiler, memory-profiler, bullet Перенес код по импорту данных из rake task в отдельный интерактор ImportData -Написал rspec тесты +Для того, чтобы лучше разобраться в коде и защитить его, написал rspec тесты и сделал рефакторинг + + ## Результаты From cc824cc270974c487e163725a5e6c1929247a838 Mon Sep 17 00:00:00 2001 From: MaksimPW Date: Wed, 26 Apr 2023 16:02:08 +0700 Subject: [PATCH 09/13] step 1: does not send unnecessary sql requests to db --- Gemfile | 3 +- Gemfile.lock | 188 ++++++++++++++------------ app/interactions/import_data.rb | 32 +++-- app/models/bus.rb | 4 +- app/models/buses_service.rb | 4 + app/models/service.rb | 3 +- bin/bootsnap | 29 ++++ bin/racc | 29 ++++ case-study.md | 47 ++++++- spec/interactions/import_data_spec.rb | 22 ++- spec/spec_helper.rb | 5 + 11 files changed, 264 insertions(+), 102 deletions(-) create mode 100644 app/models/buses_service.rb create mode 100755 bin/bootsnap create mode 100755 bin/racc diff --git a/Gemfile b/Gemfile index 884a0f5a..8b3e325b 100644 --- a/Gemfile +++ b/Gemfile @@ -8,8 +8,8 @@ gem 'pg', '>= 0.18', '< 2.0' gem 'puma', '~> 3.11' gem 'bootsnap', '>= 1.1.0', require: false gem 'activerecord-import' -gem 'oj', require: false gem 'active_interaction' +gem 'fast_jsonparser', require: false # Profiling gem 'pghero' @@ -33,6 +33,7 @@ group :test do gem 'rspec' gem 'rspec-rails' gem 'rspec-benchmark' + gem 'rspec-sqlimit' end # Windows does not include zoneinfo files, so bundle the tzinfo-data gem diff --git a/Gemfile.lock b/Gemfile.lock index 37709380..dfc760d7 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,25 +1,25 @@ GEM remote: https://rubygems.org/ specs: - actioncable (5.2.3) - actionpack (= 5.2.3) + actioncable (5.2.8.1) + actionpack (= 5.2.8.1) nio4r (~> 2.0) websocket-driver (>= 0.6.1) - actionmailer (5.2.3) - actionpack (= 5.2.3) - actionview (= 5.2.3) - activejob (= 5.2.3) + actionmailer (5.2.8.1) + actionpack (= 5.2.8.1) + actionview (= 5.2.8.1) + activejob (= 5.2.8.1) mail (~> 2.5, >= 2.5.4) rails-dom-testing (~> 2.0) - actionpack (5.2.3) - actionview (= 5.2.3) - activesupport (= 5.2.3) - rack (~> 2.0) + actionpack (5.2.8.1) + actionview (= 5.2.8.1) + activesupport (= 5.2.8.1) + rack (~> 2.0, >= 2.0.8) rack-test (>= 0.6.3) rails-dom-testing (~> 2.0) rails-html-sanitizer (~> 1.0, >= 1.0.2) - actionview (5.2.3) - activesupport (= 5.2.3) + actionview (5.2.8.1) + activesupport (= 5.2.8.1) builder (~> 3.1) erubi (~> 1.4) rails-dom-testing (~> 2.0) @@ -27,22 +27,22 @@ GEM active_interaction (4.1.0) activemodel (>= 5, < 8) activesupport (>= 5, < 8) - activejob (5.2.3) - activesupport (= 5.2.3) + activejob (5.2.8.1) + activesupport (= 5.2.8.1) globalid (>= 0.3.6) - activemodel (5.2.3) - activesupport (= 5.2.3) - activerecord (5.2.3) - activemodel (= 5.2.3) - activesupport (= 5.2.3) + activemodel (5.2.8.1) + activesupport (= 5.2.8.1) + activerecord (5.2.8.1) + activemodel (= 5.2.8.1) + activesupport (= 5.2.8.1) arel (>= 9.0) activerecord-import (1.4.1) activerecord (>= 4.2) - activestorage (5.2.3) - actionpack (= 5.2.3) - activerecord (= 5.2.3) - marcel (~> 0.3.1) - activesupport (5.2.3) + activestorage (5.2.8.1) + actionpack (= 5.2.8.1) + activerecord (= 5.2.8.1) + marcel (~> 1.0.0) + activesupport (5.2.8.1) concurrent-ruby (~> 1.0, >= 1.0.2) i18n (>= 0.7, < 2) minitest (~> 5.1) @@ -51,81 +51,94 @@ GEM benchmark-malloc (0.2.0) benchmark-perf (0.6.0) benchmark-trend (0.4.0) - bindex (0.6.0) - bootsnap (1.4.2) - msgpack (~> 1.0) - builder (3.2.3) + bindex (0.8.1) + bootsnap (1.16.0) + msgpack (~> 1.2) + builder (3.2.4) bullet (7.0.7) activesupport (>= 3.0.0) uniform_notifier (~> 1.11) - byebug (11.0.1) - concurrent-ruby (1.1.5) - crass (1.0.4) + byebug (11.1.3) + concurrent-ruby (1.2.2) + crass (1.0.6) + date (3.3.3) diff-lcs (1.5.0) - erubi (1.8.0) - ffi (1.10.0) - globalid (0.4.2) - activesupport (>= 4.2.0) - i18n (1.6.0) + erubi (1.12.0) + fast_jsonparser (0.6.0) + ffi (1.15.5) + globalid (1.1.0) + activesupport (>= 5.0) + i18n (1.12.0) concurrent-ruby (~> 1.0) listen (3.1.5) rb-fsevent (~> 0.9, >= 0.9.4) rb-inotify (~> 0.9, >= 0.9.7) ruby_dep (~> 1.2) - loofah (2.2.3) + loofah (2.20.0) crass (~> 1.0.2) nokogiri (>= 1.5.9) - mail (2.7.1) + mail (2.8.1) mini_mime (>= 0.1.1) - marcel (0.3.3) - mimemagic (~> 0.3.2) + net-imap + net-pop + net-smtp + marcel (1.0.2) memory_profiler (1.0.1) - method_source (0.9.2) - mimemagic (0.3.3) - mini_mime (1.0.1) - mini_portile2 (2.4.0) - minitest (5.11.3) - msgpack (1.2.9) - nio4r (2.3.1) - nokogiri (1.10.2) - mini_portile2 (~> 2.4.0) - oj (3.14.2) - pg (1.1.4) + method_source (1.0.0) + mini_mime (1.1.2) + mini_portile2 (2.8.1) + minitest (5.18.0) + msgpack (1.7.0) + net-imap (0.3.4) + date + net-protocol + net-pop (0.1.2) + net-protocol + net-protocol (0.2.1) + timeout + net-smtp (0.3.3) + net-protocol + nio4r (2.5.9) + nokogiri (1.13.10) + mini_portile2 (~> 2.8.0) + racc (~> 1.4) + pg (1.5.1) pghero (2.8.3) activerecord (>= 5) - puma (3.12.1) - rack (2.0.6) + puma (3.12.6) + racc (1.6.2) + rack (2.2.7) rack-mini-profiler (3.1.0) rack (>= 1.2.0) - rack-test (1.1.0) - rack (>= 1.0, < 3) - rails (5.2.3) - actioncable (= 5.2.3) - actionmailer (= 5.2.3) - actionpack (= 5.2.3) - actionview (= 5.2.3) - activejob (= 5.2.3) - activemodel (= 5.2.3) - activerecord (= 5.2.3) - activestorage (= 5.2.3) - activesupport (= 5.2.3) + rack-test (2.1.0) + rack (>= 1.3) + rails (5.2.8.1) + actioncable (= 5.2.8.1) + actionmailer (= 5.2.8.1) + actionpack (= 5.2.8.1) + actionview (= 5.2.8.1) + activejob (= 5.2.8.1) + activemodel (= 5.2.8.1) + activerecord (= 5.2.8.1) + activestorage (= 5.2.8.1) + activesupport (= 5.2.8.1) bundler (>= 1.3.0) - railties (= 5.2.3) + railties (= 5.2.8.1) sprockets-rails (>= 2.0.0) rails-dom-testing (2.0.3) activesupport (>= 4.2.0) nokogiri (>= 1.6) - rails-html-sanitizer (1.0.4) - loofah (~> 2.2, >= 2.2.2) - railties (5.2.3) - actionpack (= 5.2.3) - activesupport (= 5.2.3) + rails-html-sanitizer (1.5.0) + loofah (~> 2.19, >= 2.19.1) + railties (5.2.8.1) + actionpack (= 5.2.8.1) + activesupport (= 5.2.8.1) method_source rake (>= 0.8.7) thor (>= 0.19.0, < 2.0) - rake (12.3.2) - rb-fsevent (0.10.3) - rb-inotify (0.10.0) + rake (13.0.6) + rb-fsevent (0.11.2) + rb-inotify (0.10.1) ffi (~> 1.0) rspec (3.12.0) rspec-core (~> 3.12.0) @@ -152,19 +165,23 @@ GEM rspec-expectations (~> 3.10) rspec-mocks (~> 3.10) rspec-support (~> 3.10) + rspec-sqlimit (0.0.5) + activerecord (> 4.2, < 7.1) + rspec (~> 3.0) rspec-support (3.12.0) ruby_dep (1.5.0) - sprockets (3.7.2) + sprockets (4.2.0) concurrent-ruby (~> 1.0) - rack (> 1, < 3) - sprockets-rails (3.2.1) - actionpack (>= 4.0) - activesupport (>= 4.0) + rack (>= 2.2.4, < 4) + sprockets-rails (3.4.2) + actionpack (>= 5.2) + activesupport (>= 5.2) sprockets (>= 3.0.0) stackprof (0.2.25) - thor (0.20.3) + thor (1.2.1) thread_safe (0.3.6) - tzinfo (1.2.5) + timeout (0.3.2) + tzinfo (1.2.11) thread_safe (~> 0.1) uniform_notifier (1.16.0) web-console (3.7.0) @@ -172,9 +189,9 @@ GEM activemodel (>= 5.0) bindex (>= 0.4.0) railties (>= 5.0) - websocket-driver (0.7.0) + websocket-driver (0.7.5) websocket-extensions (>= 0.1.0) - websocket-extensions (0.1.3) + websocket-extensions (0.1.5) PLATFORMS ruby @@ -185,9 +202,9 @@ DEPENDENCIES bootsnap (>= 1.1.0) bullet byebug + fast_jsonparser listen (>= 3.0.5, < 3.2) memory_profiler - oj pg (>= 0.18, < 2.0) pghero puma (~> 3.11) @@ -196,6 +213,7 @@ DEPENDENCIES rspec rspec-benchmark rspec-rails + rspec-sqlimit stackprof tzinfo-data web-console (>= 3.3.0) @@ -204,4 +222,4 @@ RUBY VERSION ruby 2.6.3p62 BUNDLED WITH - 2.0.2 + 2.0.1 diff --git a/app/interactions/import_data.rb b/app/interactions/import_data.rb index 84dd1915..e02b6447 100644 --- a/app/interactions/import_data.rb +++ b/app/interactions/import_data.rb @@ -1,9 +1,11 @@ # frozen_string_literal: true -require 'activerecord-import' +require 'fast_jsonparser' class ImportData < ActiveInteraction::Base string :file_name + attr_reader :trips, :services, :buses, :cities + def execute start_time = Time.now @@ -23,19 +25,25 @@ def drop_data Bus.delete_all Service.delete_all Trip.delete_all - ActiveRecord::Base.connection.execute('delete from buses_services;') + BusesService.delete_all end def import_data - json = JSON.parse(File.read(file_name)) + @trips = [] + @buses = {} + @services = {} + @cities = {} - trips = [] + json = FastJsonparser.parse(File.read(file_name), symbolize_keys: false) json.each do |trip| trips << trip_new(trip) print '.' end + Service.import services.values + Bus.import buses.values, recursive: true + City.import cities.values Trip.import trips end @@ -51,18 +59,24 @@ def trip_new(trip) end def fetch_city(name) - City.find_or_create_by(name: name) + cities[name] ||= City.new(name: name) end def fetch_bus(trip) - bus = Bus.find_or_create_by(number: trip['bus']['number']) - bus.update(model: trip['bus']['model'], services: fetch_bus_services(trip)) - bus + bus = buses[trip['bus']['number']] + return bus if bus + + buses[trip['bus']['number']] = + Bus.new( + number: trip['bus']['number'], + model: trip['bus']['model'], + services: fetch_bus_services(trip) + ) end def fetch_bus_services(trip) trip['bus']['services'].map do |service| - Service.find_or_create_by(name: service) + services[service] ||= Service.new(name: service) end end diff --git a/app/models/bus.rb b/app/models/bus.rb index 1dcc54cb..5cdab4c7 100644 --- a/app/models/bus.rb +++ b/app/models/bus.rb @@ -13,7 +13,9 @@ class Bus < ApplicationRecord ].freeze has_many :trips - has_and_belongs_to_many :services, join_table: :buses_services + + has_many :buses_services + has_many :services, through: :buses_services validates :number, presence: true, uniqueness: true validates :model, inclusion: { in: MODELS } diff --git a/app/models/buses_service.rb b/app/models/buses_service.rb new file mode 100644 index 00000000..6219d44e --- /dev/null +++ b/app/models/buses_service.rb @@ -0,0 +1,4 @@ +class BusesService < ApplicationRecord + belongs_to :bus + belongs_to :service +end diff --git a/app/models/service.rb b/app/models/service.rb index 9cbb2a32..1781543c 100644 --- a/app/models/service.rb +++ b/app/models/service.rb @@ -12,7 +12,8 @@ class Service < ApplicationRecord 'Можно не печатать билет', ].freeze - has_and_belongs_to_many :buses, join_table: :buses_services + has_many :buses_services + has_many :buses, through: :buses_services validates :name, presence: true validates :name, inclusion: { in: SERVICES } diff --git a/bin/bootsnap b/bin/bootsnap new file mode 100755 index 00000000..47c8683e --- /dev/null +++ b/bin/bootsnap @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'bootsnap' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("bootsnap", "bootsnap") diff --git a/bin/racc b/bin/racc new file mode 100755 index 00000000..87e4820d --- /dev/null +++ b/bin/racc @@ -0,0 +1,29 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# +# This file was generated by Bundler. +# +# The application 'racc' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require "pathname" +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +bundle_binstub = File.expand_path("../bundle", __FILE__) + +if File.file?(bundle_binstub) + if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + load(bundle_binstub) + else + abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. +Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.") + end +end + +require "rubygems" +require "bundler/setup" + +load Gem.bin_path("racc", "racc") diff --git a/case-study.md b/case-study.md index 61f62b88..67967aca 100644 --- a/case-study.md +++ b/case-study.md @@ -6,7 +6,12 @@ Страница расписаний формируется не эффективно, механизм перезагрузки расписаний из файла занимает очень много времени(больше минуты) -Нужно исправить это и сделать механизм перезагрузки расписаний быстрее, чтобы его обработка занимала меньше одной минуты +## Задачи + + +### №1 Сократить время выполнения импорта данных + +### №2 ## Формирование метрики Для того, чтобы понимать, дают ли мои изменения положительный эффект на быстродействие программы я придумал использовать такую метрику: время исполнения rake таски и страницы с рейсами @@ -38,11 +43,47 @@ large.json отрабатывал слишком долго, не стал жд Подготовка: -Добавил в проект pghero, rack-mini-profiler, memory-profiler, bullet -Перенес код по импорту данных из rake task в отдельный интерактор ImportData +Добавил в проект +- pghero +- rack-mini-profiler +- memory-profiler +- bullet +- rspec-benchmark +- rspec-sqlimit + +Перенес код по импорту данных из rake task в отдельный интерактор `ImportData` + Для того, чтобы лучше разобраться в коде и защитить его, написал rspec тесты и сделал рефакторинг +Профилирование: + +С помощью тестов `rspec-sqlimit` смог увидеть количество запросов, которые я делаю в базу + +Как оказалось, использовать `activerecord-import` только лишь для Trip было не так корректно, так как к другим таблицам все равно было очень много запросов. + +Изменения: + +Сделал хранение всей информации в хеше. Это позволит нам не делать так много запросов SELECT к базе, чтобы найти запись. + +Использовал метод `import` из гема `activerecord-import` для других моделей: +Service, Bus, City + +Результаты: + +| |ДО |ПОСЛЕ | +|------------------|--------------------------|--------------------------| +| example.json | 0.547757 сек | 0.257931 сек | +| large.json | очень долго | 56 сек | +| medium.json | 229 сек | 10 сек | +| small.json | 30 сек | 2.990186 сек | + +Выводы: + +Нагрузка на выросла за счет того, что все данные стали хранится в хешах(MEMORY USAGE: 1067 MB) +Но я смог сократить время работы за счет уменьшения количества запросов в базу. +Так что по метрике задача №1 была выполнена +Думаю, что можно сократить и показатели по памяти, но тогда нужно будет отказаться от `activerecord-import` и стримить записи сразу в бд ## Результаты diff --git a/spec/interactions/import_data_spec.rb b/spec/interactions/import_data_spec.rb index 670e7d46..2436e4d4 100644 --- a/spec/interactions/import_data_spec.rb +++ b/spec/interactions/import_data_spec.rb @@ -1,7 +1,8 @@ require 'rails_helper' RSpec.describe ImportData do - subject { described_class.run!(file_name: 'fixtures/example.json') } + subject { described_class.run!(file_name: file_name) } + let(:file_name) { 'fixtures/example.json' } describe '#execute' do context 'common' do @@ -9,8 +10,25 @@ expect { subject } .to change { Trip.count }.by(10) .and change { City.count }.by(2) - .and change { Service.count }.by(2) + .and change { Service.count }.by(2) # 10 .and change { Bus.count }.by(1) + .and change { BusesService.count }.by(2) + end + + it 'does not send unnecessary INSERT requests to db' do + expect { subject }.not_to exceed_query_limit(5).with(/^INSERT/) + end + + it 'does not send unnecessary SELECT requests to db' do + expect { subject }.not_to exceed_query_limit(0).with(/^SELECT/) + end + + it 'does not send unnecessary DELETE requests to db' do + expect { subject }.not_to exceed_query_limit(5).with(/^DELETE/) + end + + it 'works with better performance' do + expect { subject }.to perform_under(8).us.warmup(2).times.sample(10).times end end end diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 327b58ea..fc2ba41e 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -1,3 +1,6 @@ +require 'rspec-sqlimit' +require 'rspec-benchmark' + # This file was generated by the `rails generate rspec:install` command. Conventionally, all # specs live under a `spec` directory, which RSpec adds to the `$LOAD_PATH`. # The generated `.rspec` file contains `--require spec_helper` which will cause @@ -14,6 +17,8 @@ # # See https://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration RSpec.configure do |config| + config.include RSpec::Benchmark::Matchers + # rspec-expectations config goes here. You can use an alternate # assertion/expectation library such as wrong or the stdlib/minitest # assertions if you prefer. From b947204a07f5d8249620d52700fc7fc5f280d24e Mon Sep 17 00:00:00 2001 From: MaksimPW Date: Thu, 27 Apr 2023 16:40:37 +0700 Subject: [PATCH 10/13] step 2: optimization trips view --- app/controllers/trips_controller.rb | 2 +- app/views/trips/_delimiter.html.erb | 1 - app/views/trips/_service.html.erb | 1 - app/views/trips/_services.html.erb | 6 -- app/views/trips/_trip.html.erb | 5 -- app/views/trips/index.html.erb | 18 ++++-- case-study.md | 86 ++++++++++++++++++++++++++--- 7 files changed, 92 insertions(+), 27 deletions(-) delete mode 100644 app/views/trips/_delimiter.html.erb delete mode 100644 app/views/trips/_service.html.erb delete mode 100644 app/views/trips/_services.html.erb delete mode 100644 app/views/trips/_trip.html.erb diff --git a/app/controllers/trips_controller.rb b/app/controllers/trips_controller.rb index acb38be2..ae822ef7 100644 --- a/app/controllers/trips_controller.rb +++ b/app/controllers/trips_controller.rb @@ -2,6 +2,6 @@ class TripsController < ApplicationController def index @from = City.find_by_name!(params[:from]) @to = City.find_by_name!(params[:to]) - @trips = Trip.where(from: @from, to: @to).order(:start_time) + @trips = Trip.preload(bus: :services).where(from: @from, to: @to).order(:start_time).load end end diff --git a/app/views/trips/_delimiter.html.erb b/app/views/trips/_delimiter.html.erb deleted file mode 100644 index 3f845ad0..00000000 --- a/app/views/trips/_delimiter.html.erb +++ /dev/null @@ -1 +0,0 @@ -==================================================== diff --git a/app/views/trips/_service.html.erb b/app/views/trips/_service.html.erb deleted file mode 100644 index 178ea8c0..00000000 --- a/app/views/trips/_service.html.erb +++ /dev/null @@ -1 +0,0 @@ -
  • <%= "#{service.name}" %>
    • diff --git a/app/views/trips/_services.html.erb b/app/views/trips/_services.html.erb deleted file mode 100644 index 2de639fc..00000000 --- a/app/views/trips/_services.html.erb +++ /dev/null @@ -1,6 +0,0 @@ -
  • Сервисы в автобусе:
    • -
      - <% services.each do |service| %> - <%= render "service", service: service %> - <% end %> -
    diff --git a/app/views/trips/_trip.html.erb b/app/views/trips/_trip.html.erb deleted file mode 100644 index fa1de9aa..00000000 --- a/app/views/trips/_trip.html.erb +++ /dev/null @@ -1,5 +0,0 @@ -
  • <%= "Отправление: #{trip.start_time}" %>
    • -
  • <%= "Прибытие: #{(Time.parse(trip.start_time) + trip.duration_minutes.minutes).strftime('%H:%M')}" %>
    • -
  • <%= "В пути: #{trip.duration_minutes / 60}ч. #{trip.duration_minutes % 60}мин." %>
    • -
  • <%= "Цена: #{trip.price_cents / 100}р. #{trip.price_cents % 100}коп." %>
    • -
  • <%= "Автобус: #{trip.bus.model} №#{trip.bus.number}" %>
    • diff --git a/app/views/trips/index.html.erb b/app/views/trips/index.html.erb index a60bce41..b8c906ef 100644 --- a/app/views/trips/index.html.erb +++ b/app/views/trips/index.html.erb @@ -2,15 +2,25 @@ <%= "Автобусы #{@from.name} – #{@to.name}" %>

    - <%= "В расписании #{@trips.count} рейсов" %> + <%= "В расписании #{@trips.size} рейсов" %>

    <% @trips.each do |trip| %>
      - <%= render "trip", trip: trip %> +
    • <%= "Отправление: #{trip.start_time}" %>
      • +
    • <%= "Прибытие: #{(Time.parse(trip.start_time) + trip.duration_minutes.minutes).strftime('%H:%M')}" %>
      • +
    • <%= "В пути: #{trip.duration_minutes / 60}ч. #{trip.duration_minutes % 60}мин." %>
      • +
    • <%= "Цена: #{trip.price_cents / 100}р. #{trip.price_cents % 100}коп." %>
      • +
    • <%= "Автобус: #{trip.bus.model} №#{trip.bus.number}" %>
      • + <% if trip.bus.services.present? %> - <%= render "services", services: trip.bus.services %> +
    • Сервисы в автобусе:
      • +
        + <% trip.bus.services.each do |service| %> +
      • <%= "#{service.name}" %>
        • + <% end %> +
      <% end %>
    - <%= render "delimiter" %> + ==================================================== <% end %> diff --git a/case-study.md b/case-study.md index 67967aca..1be49872 100644 --- a/case-study.md +++ b/case-study.md @@ -11,7 +11,7 @@ ### №1 Сократить время выполнения импорта данных -### №2 +### №2 Оптимизировать загрузку отображения расписаний ## Формирование метрики Для того, чтобы понимать, дают ли мои изменения положительный эффект на быстродействие программы я придумал использовать такую метрику: время исполнения rake таски и страницы с рейсами @@ -28,7 +28,7 @@ Вот какие проблемы удалось найти и решить -### Ваша находка №0 +### Шаг №0 Прогнал rake reload_json для всех файлов в fixtures, записал результаты large.json отрабатывал слишком долго, не стал ждать @@ -39,9 +39,11 @@ large.json отрабатывал слишком долго, не стал жд Поставил его, применил для импорта Trip После прогона make reload_small время работы сократилось и составило 26 сек -### Ваша находка №1 +### Шаг №1 -Подготовка: +В этом шаге решил сконцентрироваться на загрузке данных из json в базу + +#### Подготовка: Добавил в проект - pghero @@ -55,20 +57,20 @@ large.json отрабатывал слишком долго, не стал жд Для того, чтобы лучше разобраться в коде и защитить его, написал rspec тесты и сделал рефакторинг -Профилирование: +#### Профилирование: С помощью тестов `rspec-sqlimit` смог увидеть количество запросов, которые я делаю в базу Как оказалось, использовать `activerecord-import` только лишь для Trip было не так корректно, так как к другим таблицам все равно было очень много запросов. -Изменения: +#### Изменения: Сделал хранение всей информации в хеше. Это позволит нам не делать так много запросов SELECT к базе, чтобы найти запись. Использовал метод `import` из гема `activerecord-import` для других моделей: Service, Bus, City -Результаты: +#### Результаты: | |ДО |ПОСЛЕ | |------------------|--------------------------|--------------------------| @@ -77,14 +79,80 @@ Service, Bus, City | medium.json | 229 сек | 10 сек | | small.json | 30 сек | 2.990186 сек | -Выводы: +#### Выводы: -Нагрузка на выросла за счет того, что все данные стали хранится в хешах(MEMORY USAGE: 1067 MB) +Нагрузка на память выросла за счет того, что все данные стали хранится в хешах(MEMORY USAGE: 1067 MB) Но я смог сократить время работы за счет уменьшения количества запросов в базу. Так что по метрике задача №1 была выполнена Думаю, что можно сократить и показатели по памяти, но тогда нужно будет отказаться от `activerecord-import` и стримить записи сразу в бд +### Шаг №3 + +В этом шаге решил сконцентрироваться на оптимизации загрузки страницы с рейсами + +Импортировал данные из файла large.json в базу + +#### 1. + +Страница с рейсами загрузилась за 24158.2 ms +`bullet` сразу дал подсказку 'USE eager loading detected' для `app/views/trips/_trip.html.erb` + +Результаты rack-mini-profiler так же указали на то, что здесь происходит очень много запросов на получении информации по каждому автобусу +``` +Rendering: trips/index.html.erb 10686.2 +18.8 650 sql 1815.9 +``` + +Добавил `includes([:bus])` в запрос на получение @trips + +#### 2. + +Дальше `bullet` стал жаловаться на такую же проблему N+1 и для `services` + +Изменил запрос на получение @trips, чтобы дополнительно загружать еще и services: +``` +@trips = Trip.preload(bus: :services).where(from: @from, to: @to).order(:start_time).load +``` + +После этого ActiveRecord больше не был главной точкой роста + +Результаты rack-mini-profiler: +``` +Rendering: trips/index.html.erb 6081.0 +2326.9 1 sql 9.3 +``` + +Логи Rails: +``` +Completed 200 OK in 17511ms (Views: 15219.8ms | ActiveRecord: 100.8ms) +``` + +#### 3. + +По логам и результатам профилировщика rack-mini-profiler можно было заметить, что теперь основное время тратилось на загрузку вьюх + +Решил избавиться от partials и перенести весь код из них в trips/index.html.erb +После изменений страница стала загружаться быстрее: +``` +Completed 200 OK in 6027ms (Views: 3736.2ms | ActiveRecord: 56.0ms) +``` + +#### 4. + +Нашел в trips/index.html.erb вызов count, заменил его на size +Время загрузки ActiveRecord изменилось в лучшую сторону: +``` +Completed 200 OK in 6313ms (Views: 4103.4ms | ActiveRecord: 43.3ms) +``` + +#### Результаты: + +|ДО |ПОСЛЕ | +|--------------------------|--------------------------| +| 24158.2 ms | 6313.3 ms | + +#### Выводы: + + ## Результаты ### rake reload_json From fffa2aa97c4787105ff6e96fca82aedf48fbd0fb Mon Sep 17 00:00:00 2001 From: MaksimPW Date: Thu, 27 Apr 2023 18:15:44 +0700 Subject: [PATCH 11/13] step 2: use partials with collection --- app/views/trips/_delimiter.html | 1 + app/views/trips/_service.html.erb | 1 + app/views/trips/_trip.html.erb | 12 ++++++++++++ app/views/trips/index.html.erb | 20 +------------------- case-study.md | 16 +++++++++++++++- 5 files changed, 30 insertions(+), 20 deletions(-) create mode 100644 app/views/trips/_delimiter.html create mode 100644 app/views/trips/_service.html.erb create mode 100644 app/views/trips/_trip.html.erb diff --git a/app/views/trips/_delimiter.html b/app/views/trips/_delimiter.html new file mode 100644 index 00000000..18c9291f --- /dev/null +++ b/app/views/trips/_delimiter.html @@ -0,0 +1 @@ +==================================================== \ No newline at end of file diff --git a/app/views/trips/_service.html.erb b/app/views/trips/_service.html.erb new file mode 100644 index 00000000..a2a4af0f --- /dev/null +++ b/app/views/trips/_service.html.erb @@ -0,0 +1 @@ +
  • <%= "#{service.name}" %>
    • \ No newline at end of file diff --git a/app/views/trips/_trip.html.erb b/app/views/trips/_trip.html.erb new file mode 100644 index 00000000..6e51b686 --- /dev/null +++ b/app/views/trips/_trip.html.erb @@ -0,0 +1,12 @@ +
      +
    • <%= "Отправление: #{trip.start_time}" %>
      • +
    • <%= "Прибытие: #{(Time.parse(trip.start_time) + trip.duration_minutes.minutes).strftime('%H:%M')}" %>
      • +
    • <%= "В пути: #{trip.duration_minutes / 60}ч. #{trip.duration_minutes % 60}мин." %>
      • +
    • <%= "Цена: #{trip.price_cents / 100}р. #{trip.price_cents % 100}коп." %>
      • +
    • <%= "Автобус: #{trip.bus.model} №#{trip.bus.number}" %>
      • + +
    • Сервисы в автобусе:
      • +
        + <%= render partial: 'service', collection: trip.bus.services %> +
      +
    diff --git a/app/views/trips/index.html.erb b/app/views/trips/index.html.erb index b8c906ef..648aa4b7 100644 --- a/app/views/trips/index.html.erb +++ b/app/views/trips/index.html.erb @@ -5,22 +5,4 @@ <%= "В расписании #{@trips.size} рейсов" %> -<% @trips.each do |trip| %> -
      -
    • <%= "Отправление: #{trip.start_time}" %>
      • -
    • <%= "Прибытие: #{(Time.parse(trip.start_time) + trip.duration_minutes.minutes).strftime('%H:%M')}" %>
      • -
    • <%= "В пути: #{trip.duration_minutes / 60}ч. #{trip.duration_minutes % 60}мин." %>
      • -
    • <%= "Цена: #{trip.price_cents / 100}р. #{trip.price_cents % 100}коп." %>
      • -
    • <%= "Автобус: #{trip.bus.model} №#{trip.bus.number}" %>
      • - - <% if trip.bus.services.present? %> -
    • Сервисы в автобусе:
      • -
        - <% trip.bus.services.each do |service| %> -
      • <%= "#{service.name}" %>
        • - <% end %> -
      - <% end %> -
    - ==================================================== -<% end %> +<%= render partial: 'trip', collection: @trips, spacer_template: 'delimiter' %> diff --git a/case-study.md b/case-study.md index 1be49872..67245cb3 100644 --- a/case-study.md +++ b/case-study.md @@ -144,6 +144,16 @@ Completed 200 OK in 6027ms (Views: 3736.2ms | ActiveRecord: 56.0ms) Completed 200 OK in 6313ms (Views: 4103.4ms | ActiveRecord: 43.3ms) ``` + +#### 5. + +Вернул обратно partials, но сделал их вызов с указанием коллекции +Это сделало код более читабельным, но при этом мы не потеряли в производительности + +#### 6. + + + #### Результаты: |ДО |ПОСЛЕ | @@ -164,7 +174,11 @@ Completed 200 OK in 6313ms (Views: 4103.4ms | ActiveRecord: 43.3ms) | medium.json | 229 сек | - | | small.json | 30 сек | - | -### Заметки + +### Наблюдения + +В рабочем проекте используем pghero как хороший инструмент для выявления медленных запросов, подсказок по индексам, занятой памяти +В этом проекте pghero показывает все зеленым. Может быть очень мало запросов #### Совет: как посчитать кол-во строк в файле ``` From 5a4b58105b285e8737f557aaa2b0c262f0eb1526 Mon Sep 17 00:00:00 2001 From: MaksimPW Date: Thu, 27 Apr 2023 19:26:23 +0700 Subject: [PATCH 12/13] step 4: upgrade ruby && rails --- Gemfile | 4 +- Gemfile.lock | 175 +++++++++++++++++++----------------- bin/bootsnap | 8 +- bin/bundle | 40 +++++---- bin/byebug | 8 +- bin/htmldiff | 8 +- bin/ldiff | 8 +- bin/listen | 8 +- bin/nokogiri | 8 +- bin/puma | 8 +- bin/pumactl | 8 +- bin/racc | 8 +- bin/rackup | 8 +- bin/rails | 8 +- bin/rake | 8 +- bin/rspec | 8 +- bin/ruby-memory-profiler | 8 +- bin/sprockets | 8 +- bin/stackprof | 8 +- bin/stackprof-flamegraph.pl | 8 +- bin/stackprof-gprof2dot.py | 8 +- bin/thor | 8 +- case-study.md | 43 ++++++--- 23 files changed, 205 insertions(+), 209 deletions(-) diff --git a/Gemfile b/Gemfile index 8b3e325b..1e448ca6 100644 --- a/Gemfile +++ b/Gemfile @@ -1,9 +1,9 @@ source 'https://rubygems.org' git_source(:github) { |repo| "https://github.com/#{repo}.git" } -ruby '2.6.3' +ruby '3.2.2' -gem 'rails', '~> 5.2.3' +gem 'rails', '~> 6.0' gem 'pg', '>= 0.18', '< 2.0' gem 'puma', '~> 3.11' gem 'bootsnap', '>= 1.1.0', require: false diff --git a/Gemfile.lock b/Gemfile.lock index dfc760d7..baae26ac 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,53 +1,70 @@ GEM remote: https://rubygems.org/ specs: - actioncable (5.2.8.1) - actionpack (= 5.2.8.1) + actioncable (6.1.7.3) + actionpack (= 6.1.7.3) + activesupport (= 6.1.7.3) nio4r (~> 2.0) websocket-driver (>= 0.6.1) - actionmailer (5.2.8.1) - actionpack (= 5.2.8.1) - actionview (= 5.2.8.1) - activejob (= 5.2.8.1) + actionmailbox (6.1.7.3) + actionpack (= 6.1.7.3) + activejob (= 6.1.7.3) + activerecord (= 6.1.7.3) + activestorage (= 6.1.7.3) + activesupport (= 6.1.7.3) + mail (>= 2.7.1) + actionmailer (6.1.7.3) + actionpack (= 6.1.7.3) + actionview (= 6.1.7.3) + activejob (= 6.1.7.3) + activesupport (= 6.1.7.3) mail (~> 2.5, >= 2.5.4) rails-dom-testing (~> 2.0) - actionpack (5.2.8.1) - actionview (= 5.2.8.1) - activesupport (= 5.2.8.1) - rack (~> 2.0, >= 2.0.8) + actionpack (6.1.7.3) + actionview (= 6.1.7.3) + activesupport (= 6.1.7.3) + rack (~> 2.0, >= 2.0.9) rack-test (>= 0.6.3) rails-dom-testing (~> 2.0) - rails-html-sanitizer (~> 1.0, >= 1.0.2) - actionview (5.2.8.1) - activesupport (= 5.2.8.1) + rails-html-sanitizer (~> 1.0, >= 1.2.0) + actiontext (6.1.7.3) + actionpack (= 6.1.7.3) + activerecord (= 6.1.7.3) + activestorage (= 6.1.7.3) + activesupport (= 6.1.7.3) + nokogiri (>= 1.8.5) + actionview (6.1.7.3) + activesupport (= 6.1.7.3) builder (~> 3.1) erubi (~> 1.4) rails-dom-testing (~> 2.0) - rails-html-sanitizer (~> 1.0, >= 1.0.3) - active_interaction (4.1.0) - activemodel (>= 5, < 8) - activesupport (>= 5, < 8) - activejob (5.2.8.1) - activesupport (= 5.2.8.1) + rails-html-sanitizer (~> 1.1, >= 1.2.0) + active_interaction (5.2.0) + activemodel (>= 5.2, < 8) + activesupport (>= 5.2, < 8) + activejob (6.1.7.3) + activesupport (= 6.1.7.3) globalid (>= 0.3.6) - activemodel (5.2.8.1) - activesupport (= 5.2.8.1) - activerecord (5.2.8.1) - activemodel (= 5.2.8.1) - activesupport (= 5.2.8.1) - arel (>= 9.0) + activemodel (6.1.7.3) + activesupport (= 6.1.7.3) + activerecord (6.1.7.3) + activemodel (= 6.1.7.3) + activesupport (= 6.1.7.3) activerecord-import (1.4.1) activerecord (>= 4.2) - activestorage (5.2.8.1) - actionpack (= 5.2.8.1) - activerecord (= 5.2.8.1) - marcel (~> 1.0.0) - activesupport (5.2.8.1) + activestorage (6.1.7.3) + actionpack (= 6.1.7.3) + activejob (= 6.1.7.3) + activerecord (= 6.1.7.3) + activesupport (= 6.1.7.3) + marcel (~> 1.0) + mini_mime (>= 1.1.0) + activesupport (6.1.7.3) concurrent-ruby (~> 1.0, >= 1.0.2) - i18n (>= 0.7, < 2) - minitest (~> 5.1) - tzinfo (~> 1.1) - arel (9.0.0) + i18n (>= 1.6, < 2) + minitest (>= 5.1) + tzinfo (~> 2.0) + zeitwerk (~> 2.3) benchmark-malloc (0.2.0) benchmark-perf (0.6.0) benchmark-trend (0.4.0) @@ -68,12 +85,11 @@ GEM ffi (1.15.5) globalid (1.1.0) activesupport (>= 5.0) - i18n (1.12.0) + i18n (1.13.0) concurrent-ruby (~> 1.0) - listen (3.1.5) + listen (3.0.8) rb-fsevent (~> 0.9, >= 0.9.4) rb-inotify (~> 0.9, >= 0.9.7) - ruby_dep (~> 1.2) loofah (2.20.0) crass (~> 1.0.2) nokogiri (>= 1.5.9) @@ -86,7 +102,6 @@ GEM memory_profiler (1.0.1) method_source (1.0.0) mini_mime (1.1.2) - mini_portile2 (2.8.1) minitest (5.18.0) msgpack (1.7.0) net-imap (0.3.4) @@ -99,12 +114,11 @@ GEM net-smtp (0.3.3) net-protocol nio4r (2.5.9) - nokogiri (1.13.10) - mini_portile2 (~> 2.8.0) + nokogiri (1.14.3-x86_64-darwin) racc (~> 1.4) - pg (1.5.1) - pghero (2.8.3) - activerecord (>= 5) + pg (1.5.2) + pghero (3.3.3) + activerecord (>= 6) puma (3.12.6) racc (1.6.2) rack (2.2.7) @@ -112,30 +126,32 @@ GEM rack (>= 1.2.0) rack-test (2.1.0) rack (>= 1.3) - rails (5.2.8.1) - actioncable (= 5.2.8.1) - actionmailer (= 5.2.8.1) - actionpack (= 5.2.8.1) - actionview (= 5.2.8.1) - activejob (= 5.2.8.1) - activemodel (= 5.2.8.1) - activerecord (= 5.2.8.1) - activestorage (= 5.2.8.1) - activesupport (= 5.2.8.1) - bundler (>= 1.3.0) - railties (= 5.2.8.1) + rails (6.1.7.3) + actioncable (= 6.1.7.3) + actionmailbox (= 6.1.7.3) + actionmailer (= 6.1.7.3) + actionpack (= 6.1.7.3) + actiontext (= 6.1.7.3) + actionview (= 6.1.7.3) + activejob (= 6.1.7.3) + activemodel (= 6.1.7.3) + activerecord (= 6.1.7.3) + activestorage (= 6.1.7.3) + activesupport (= 6.1.7.3) + bundler (>= 1.15.0) + railties (= 6.1.7.3) sprockets-rails (>= 2.0.0) rails-dom-testing (2.0.3) activesupport (>= 4.2.0) nokogiri (>= 1.6) rails-html-sanitizer (1.5.0) loofah (~> 2.19, >= 2.19.1) - railties (5.2.8.1) - actionpack (= 5.2.8.1) - activesupport (= 5.2.8.1) + railties (6.1.7.3) + actionpack (= 6.1.7.3) + activesupport (= 6.1.7.3) method_source - rake (>= 0.8.7) - thor (>= 0.19.0, < 2.0) + rake (>= 12.2) + thor (~> 1.0) rake (13.0.6) rb-fsevent (0.11.2) rb-inotify (0.10.1) @@ -157,19 +173,18 @@ GEM rspec-mocks (3.12.5) diff-lcs (>= 1.2.0, < 2.0) rspec-support (~> 3.12.0) - rspec-rails (5.1.2) - actionpack (>= 5.2) - activesupport (>= 5.2) - railties (>= 5.2) - rspec-core (~> 3.10) - rspec-expectations (~> 3.10) - rspec-mocks (~> 3.10) - rspec-support (~> 3.10) + rspec-rails (6.0.1) + actionpack (>= 6.1) + activesupport (>= 6.1) + railties (>= 6.1) + rspec-core (~> 3.11) + rspec-expectations (~> 3.11) + rspec-mocks (~> 3.11) + rspec-support (~> 3.11) rspec-sqlimit (0.0.5) activerecord (> 4.2, < 7.1) rspec (~> 3.0) rspec-support (3.12.0) - ruby_dep (1.5.0) sprockets (4.2.0) concurrent-ruby (~> 1.0) rack (>= 2.2.4, < 4) @@ -179,22 +194,22 @@ GEM sprockets (>= 3.0.0) stackprof (0.2.25) thor (1.2.1) - thread_safe (0.3.6) timeout (0.3.2) - tzinfo (1.2.11) - thread_safe (~> 0.1) + tzinfo (2.0.6) + concurrent-ruby (~> 1.0) uniform_notifier (1.16.0) - web-console (3.7.0) - actionview (>= 5.0) - activemodel (>= 5.0) + web-console (4.2.0) + actionview (>= 6.0.0) + activemodel (>= 6.0.0) bindex (>= 0.4.0) - railties (>= 5.0) + railties (>= 6.0.0) websocket-driver (0.7.5) websocket-extensions (>= 0.1.0) websocket-extensions (0.1.5) + zeitwerk (2.6.7) PLATFORMS - ruby + x86_64-darwin-22 DEPENDENCIES active_interaction @@ -209,7 +224,7 @@ DEPENDENCIES pghero puma (~> 3.11) rack-mini-profiler - rails (~> 5.2.3) + rails (~> 6.0) rspec rspec-benchmark rspec-rails @@ -219,7 +234,7 @@ DEPENDENCIES web-console (>= 3.3.0) RUBY VERSION - ruby 2.6.3p62 + ruby 3.2.2p53 BUNDLED WITH - 2.0.1 + 2.4.12 diff --git a/bin/bootsnap b/bin/bootsnap index 47c8683e..0f5dd65c 100755 --- a/bin/bootsnap +++ b/bin/bootsnap @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/bundle b/bin/bundle index 524dfd3f..42c7fd7c 100755 --- a/bin/bundle +++ b/bin/bundle @@ -11,7 +11,7 @@ require "rubygems" m = Module.new do - module_function + module_function def invoked_as_script? File.expand_path($0) == File.expand_path(__FILE__) @@ -31,7 +31,7 @@ m = Module.new do bundler_version = a end next unless a =~ /\A--bundler(?:[= ](#{Gem::Version::VERSION_PATTERN}))?\z/ - bundler_version = $1 || ">= 0.a" + bundler_version = $1 update_index = i end bundler_version @@ -41,13 +41,13 @@ m = Module.new do gemfile = ENV["BUNDLE_GEMFILE"] return gemfile if gemfile && !gemfile.empty? - File.expand_path("../../Gemfile", __FILE__) + File.expand_path("../Gemfile", __dir__) end def lockfile lockfile = case File.basename(gemfile) - when "gems.rb" then gemfile.sub(/\.rb$/, gemfile) + when "gems.rb" then gemfile.sub(/\.rb$/, ".locked") else "#{gemfile}.lock" end File.expand_path(lockfile) @@ -60,33 +60,37 @@ m = Module.new do Regexp.last_match(1) end - def bundler_version - @bundler_version ||= begin - env_var_version || cli_arg_version || - lockfile_version || "#{Gem::Requirement.default}.a" - end + def bundler_requirement + @bundler_requirement ||= + env_var_version || + cli_arg_version || + bundler_requirement_for(lockfile_version) + end + + def bundler_requirement_for(version) + return "#{Gem::Requirement.default}.a" unless version + + bundler_gem_version = Gem::Version.new(version) + + bundler_gem_version.approximate_recommendation end def load_bundler! ENV["BUNDLE_GEMFILE"] ||= gemfile - # must dup string for RG < 1.8 compatibility - activate_bundler(bundler_version.dup) + activate_bundler end - def activate_bundler(bundler_version) - if Gem::Version.correct?(bundler_version) && Gem::Version.new(bundler_version).release < Gem::Version.new("2.0") - bundler_version = "< 2" - end + def activate_bundler gem_error = activation_error_handling do - gem "bundler", bundler_version + gem "bundler", bundler_requirement end return if gem_error.nil? require_error = activation_error_handling do require "bundler/version" end - return if require_error.nil? && Gem::Requirement.new(bundler_version).satisfied_by?(Gem::Version.new(Bundler::VERSION)) - warn "Activating bundler (#{bundler_version}) failed:\n#{gem_error.message}\n\nTo install the version of bundler this project requires, run `gem install bundler -v '#{bundler_version}'`" + return if require_error.nil? && Gem::Requirement.new(bundler_requirement).satisfied_by?(Gem::Version.new(Bundler::VERSION)) + warn "Activating bundler (#{bundler_requirement}) failed:\n#{gem_error.message}\n\nTo install the version of bundler this project requires, run `gem install bundler -v '#{bundler_requirement}'`" exit 42 end diff --git a/bin/byebug b/bin/byebug index 16f20bbd..abc90dbf 100755 --- a/bin/byebug +++ b/bin/byebug @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/htmldiff b/bin/htmldiff index 091820c9..0aeaec87 100755 --- a/bin/htmldiff +++ b/bin/htmldiff @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/ldiff b/bin/ldiff index 073e19f2..8173edec 100755 --- a/bin/ldiff +++ b/bin/ldiff @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/listen b/bin/listen index 3dd36a7e..613171d2 100755 --- a/bin/listen +++ b/bin/listen @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/nokogiri b/bin/nokogiri index b22a1a0a..c00ec262 100755 --- a/bin/nokogiri +++ b/bin/nokogiri @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/puma b/bin/puma index 880935b2..01a92a32 100755 --- a/bin/puma +++ b/bin/puma @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/pumactl b/bin/pumactl index b698e6e9..c93cff21 100755 --- a/bin/pumactl +++ b/bin/pumactl @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/racc b/bin/racc index 87e4820d..81900158 100755 --- a/bin/racc +++ b/bin/racc @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/rackup b/bin/rackup index 3ac4a5a7..0af6fafd 100755 --- a/bin/rackup +++ b/bin/rackup @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/rails b/bin/rails index 7fd59cc7..a93ac1a5 100755 --- a/bin/rails +++ b/bin/rails @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/rake b/bin/rake index 9275675e..4eb7d7bf 100755 --- a/bin/rake +++ b/bin/rake @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/rspec b/bin/rspec index a6c78521..cb53ebe5 100755 --- a/bin/rspec +++ b/bin/rspec @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/ruby-memory-profiler b/bin/ruby-memory-profiler index b13a4b03..4f90dde1 100755 --- a/bin/ruby-memory-profiler +++ b/bin/ruby-memory-profiler @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/sprockets b/bin/sprockets index 9f75aa74..0068cd75 100755 --- a/bin/sprockets +++ b/bin/sprockets @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/stackprof b/bin/stackprof index cfc10350..17c181e7 100755 --- a/bin/stackprof +++ b/bin/stackprof @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/stackprof-flamegraph.pl b/bin/stackprof-flamegraph.pl index f6212797..517079e8 100755 --- a/bin/stackprof-flamegraph.pl +++ b/bin/stackprof-flamegraph.pl @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/stackprof-gprof2dot.py b/bin/stackprof-gprof2dot.py index e72dd002..bf8007fa 100755 --- a/bin/stackprof-gprof2dot.py +++ b/bin/stackprof-gprof2dot.py @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/bin/thor b/bin/thor index 71bfaeae..ec401151 100755 --- a/bin/thor +++ b/bin/thor @@ -8,14 +8,12 @@ # this file is here to facilitate running it. # -require "pathname" -ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", - Pathname.new(__FILE__).realpath) +ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) -bundle_binstub = File.expand_path("../bundle", __FILE__) +bundle_binstub = File.expand_path("bundle", __dir__) if File.file?(bundle_binstub) - if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/ + if File.read(bundle_binstub, 300).include?("This file was generated by Bundler") load(bundle_binstub) else abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run. diff --git a/case-study.md b/case-study.md index 67245cb3..a73b2b5d 100644 --- a/case-study.md +++ b/case-study.md @@ -72,6 +72,7 @@ Service, Bus, City #### Результаты: +Импорт данных: | |ДО |ПОСЛЕ | |------------------|--------------------------|--------------------------| | example.json | 0.547757 сек | 0.257931 сек | @@ -144,36 +145,50 @@ Completed 200 OK in 6027ms (Views: 3736.2ms | ActiveRecord: 56.0ms) Completed 200 OK in 6313ms (Views: 4103.4ms | ActiveRecord: 43.3ms) ``` - #### 5. Вернул обратно partials, но сделал их вызов с указанием коллекции Это сделало код более читабельным, но при этом мы не потеряли в производительности -#### 6. - - - #### Результаты: +Загрузка страницы с расписанием: |ДО |ПОСЛЕ | |--------------------------|--------------------------| | 24158.2 ms | 6313.3 ms | -#### Выводы: +#### Выводы +Страница с расписаниями стала намного быстрее загружаться, но 6 сек - это все равно очень большое время +Не думаю, что людям нужно сразу отображать всю информацию по расписанию, так как на странице загружается сразу 1004 рейса +В таком случае можно сделать пагинацию или последовательную загрузку данных по рейсам -## Результаты +### Шаг №4 -### rake reload_json +В этом шаге решил обновить ruby и rails, чтобы посмотреть как измениться производительность -| |ДО |ПОСЛЕ | -|------------------|--------------------------|--------------------------| -| example.json | 0.547757 | - | -| large.json | очень долго | - | -| medium.json | 229 сек | - | -| small.json | 30 сек | - | +Обновил ruby с 2.6.3 до 3.2.2 +Rails с 5.2.3 до ~> 6.0 + +#### Результаты + +Импорт данных: +| |ДО |ПОСЛЕ ПЕРВОГО ШАГА |ПОСЛЕ ОБНОВЛЕНИЯ RUBY | +|------------------|--------------------------|--------------------------|----------------------| +| example.json | 0.547757 сек | 0.257931 сек | 0.221117 сек | +| large.json | очень долго | 56 сек | 45.511732 сек | +| medium.json | 229 сек | 10 сек | 7.469367 сек | +| small.json | 30 сек | 2.990186 сек | 2.487498 сек | + +Загрузка страницы с расписанием: +``` +Completed 200 OK in 7124ms (Views: 5059.9ms | ActiveRecord: 39.5ms | Allocations: 6974902) +``` + +#### Выводы: +Не забывайте держать версию ruby в актуальном состоянии +Зачастую обновить язык - это один из самых эффективных способов повысить производительность ### Наблюдения From 35ca11e2f312a402ff978b0f0556a1efea511ac8 Mon Sep 17 00:00:00 2001 From: MaksimPW Date: Thu, 27 Apr 2023 19:31:44 +0700 Subject: [PATCH 13/13] update readme --- Readme.md | 13 +++++++------ case-study.md | 21 +++++++++------------ 2 files changed, 16 insertions(+), 18 deletions(-) diff --git a/Readme.md b/Readme.md index faf4e797..29f877de 100644 --- a/Readme.md +++ b/Readme.md @@ -3,14 +3,15 @@ В этом задании вам предлагается оптимизировать учебное `rails`-приложение. Для запуска потребуется: -- `ruby 2.6.3` -- `postgres` +- `ruby 3.2.0` +- `docker` Запуск и использование: - `bundle install` - `bin/setup` +- `docker-compose up` - `rails s` -- `open http://localhost:3000/автобусы/Самара/Москва` +- `make open` ## Описание учебного приложения Зайдя на страницу `автобусы/Самара/Москва` вы увидите расписание автобусов по этому направлению. @@ -43,11 +44,11 @@ Нужно найти и устранить проблемы, замедляющие формирование этих страниц. Попробуйте воспользоваться -- [ ] `rack-mini-profiler` +- [x] `rack-mini-profiler` - [ ] `rails panel` -- [ ] `bullet` +- [x] `bullet` - [ ] `explain` запросов -- [ ] `rspec-sqlimit` +- [x] `rspec-sqlimit` ### Сдача задания `PR` в этот репозиторий с кодом и case-study наподобие первых двух недель. На этот раз шаблона нет, законспектируйте ваш процесс оптимизации в свободной форме. diff --git a/case-study.md b/case-study.md index a73b2b5d..90588507 100644 --- a/case-study.md +++ b/case-study.md @@ -8,7 +8,6 @@ ## Задачи - ### №1 Сократить время выполнения импорта данных ### №2 Оптимизировать загрузку отображения расписаний @@ -16,17 +15,15 @@ ## Формирование метрики Для того, чтобы понимать, дают ли мои изменения положительный эффект на быстродействие программы я придумал использовать такую метрику: время исполнения rake таски и страницы с рейсами -## Feedback-Loop -Для того, чтобы иметь возможность быстро проверять гипотезы я выстроил эффективный `feedback-loop`, который позволил мне получать обратную связь по эффективности сделанных изменений за N: - -- пока не выстроил - ## Вникаем в детали системы, чтобы найти главные точки роста Для того, чтобы найти "точки роста" для оптимизации я воспользовался: -- пока ничем не воспользовался - -Вот какие проблемы удалось найти и решить +- pghero +- rack-mini-profiler +- memory-profiler +- bullet +- rspec-benchmark +- rspec-sqlimit ### Шаг №0 @@ -193,18 +190,18 @@ Completed 200 OK in 7124ms (Views: 5059.9ms | ActiveRecord: 39.5ms | Allocations ### Наблюдения В рабочем проекте используем pghero как хороший инструмент для выявления медленных запросов, подсказок по индексам, занятой памяти -В этом проекте pghero показывает все зеленым. Может быть очень мало запросов +В этом проекте pghero показывает все зеленым. Пробовал делать много запросов - все равно его все устраивает. #### Совет: как посчитать кол-во строк в файле ``` wc -l data_large.rb # (3250940) total line count ``` -#### Совет: как создать меньший файл из большего, оставив перевые N строк +#### Совет: как создать меньший файл из большего, оставив первые N строк ``` head -n N data_large.txt > dataN.txt ``` ## Защита от регрессии производительности -Пока ничего не сделано \ No newline at end of file +Были написаны performace тесты на количество запросов в базу и скорость выполнения задачи \ No newline at end of file