ruby web crawler

crawl sites with cli

command 1 links

bundle exec ruby main.rb links ... bundle exec ruby main.rb links https://foo.com/

command 2 scrape

bundle exec ruby main.rb scrape <(optional) class> bundle exec ruby main.rb scrape bundle exec ruby mian.rb scrape -class classname does not have to be provided but if looking for all of tag type name and no tag filter specify with "-tag" after

if neither tag nor class is provided defaults to parsing the full html page