Merge pull request #986 from jmerle/update-checker

Automatically check for documentation updates

Author: jmerle

.github/CONTRIBUTING.md

@@ -61,7 +61,7 @@ In addition to the [guidelines for contributing code](#contributing-code-and-fea
 
 Please don't submit a pull request updating the version number of a documentation, unless a change is required in the scraper and you've verified that it works.
 
-To ask that an existing documentation be updated, please use the [Trello board](https://trello.com/c/2B0hmW7M/52-request-updates-here).
+To ask that an existing documentation be updated, first check the last two [Documentation versions reports](https://github.com/freeCodeCamp/devdocs/issues?utf8=%E2%9C%93&q=Documentation+versions+report+is%3Aissue+author%3Adevdocs-bot+sort%3Aupdated-desc). Only create an issue if the documentation has been wrongly marked as up-to-date.
 
 ## Coding conventions
 

.travis.yml

@@ -6,6 +6,10 @@ before_script:
   - gem update --system
   - gem install bundler
 
+script:
+  - if [ "$TRAVIS_EVENT_TYPE" != "cron" ]; then bundle exec rake; fi
+  - if [ "$TRAVIS_EVENT_TYPE" = "cron" ]; then bundle exec thor updates:check --github-token $GH_TOKEN --upload; fi
+
 deploy:
   provider: heroku
   app: devdocs

Gemfile

@@ -42,6 +42,7 @@ group :docs do
   gem 'unix_utils', require: false
   gem 'tty-pager', require: false
   gem 'net-sftp', '>= 2.1.3.rc2', require: false
+  gem 'terminal-table', require: false
 end
 
 group :test do

Gemfile.lock

@@ -104,6 +104,8 @@ GEM
       unicode-display_width (~> 1.4.0)
       unicode_utils (~> 1.4.0)
     strings-ansi (0.1.0)
+    terminal-table (1.8.0)
+      unicode-display_width (~> 1.1, >= 1.1.1)
     thin (1.7.2)
       daemons (~> 1.0, >= 1.0.9)
       eventmachine (~> 1.0, >= 1.0.4)
@@ -157,6 +159,7 @@ DEPENDENCIES
   sinatra-contrib
   sprockets
   sprockets-helpers
+  terminal-table
   sprockets-sass
   thin
   thor

docs/adding-docs.md

@@ -17,6 +17,7 @@ Adding a documentation may look like a daunting task but once you get the hang o
 10. To add syntax highlighting or execute custom JavaScript on the pages, create a file in the `assets/javascripts/views/pages/` directory (take a look at the other files to see how it works).
 11. Add the documentation's icon in the `public/icons/docs/[my_doc]/` directory, in both 16x16 and 32x32-pixels formats. It'll be added to the icon spritesheet after your pull request is merged.
 12. Add the documentation's copyright details to the list in `assets/javascripts/templates/pages/about_tmpl.coffee`. This is the data shown in the table on the [about](https://devdocs.io/about) page, and is ordered alphabetically. Simply copying an existing item, placing it in the right slot and updating the values to match the new scraper will do the job.
+13. Ensure `thor updates:check [my_doc]` shows the correct latest version.
 
 If the documentation includes more than a few hundreds pages and is available for download, try to scrape it locally (e.g. using `FileScraper`). It'll make the development process much faster and avoids putting too much load on the source site. (It's not a problem if your scraper is coupled to your local setup, just explain how it works in your pull request.)
 

docs/scraper-reference.md

@@ -184,3 +184,44 @@ More information about how filters work is available on the [Filter Reference](.
     Overrides the `:title` option for the root page only.
 
   _Note: this filter is disabled by default._
+
+## Keeping scrapers up-to-date
+
+In order to keep scrapers up-to-date the `get_latest_version(opts)` method should be overridden. If `self.release` is defined, this should return the latest version of the documentation. If `self.release` is not defined, it should return the Epoch time when the documentation was last modified. If the documentation will never change, simply return `1.0.0`. The result of this method is periodically reported in a "Documentation versions report" issue which helps maintainers keep track of outdated documentations.
+
+To make life easier, there are a few utility methods that you can use in `get_latest_version`:
+* `fetch(url, opts)`
+
+  Makes a GET request to the url and returns the response body.
+
+  Example: [lib/docs/scrapers/bash.rb](../lib/docs/scrapers/bash.rb)
+* `fetch_doc(url, opts)`
+
+  Makes a GET request to the url and returns the HTML body converted to a Nokogiri document.
+
+  Example: [lib/docs/scrapers/git.rb](../lib/docs/scrapers/git.rb)
+* `fetch_json(url, opts)`
+
+  Makes a GET request to the url and returns the JSON body converted to a dictionary.
+
+  Example: [lib/docs/scrapers/mdn/mdn.rb](../lib/docs/scrapers/mdn/mdn.rb)
+* `get_npm_version(package, opts)`
+
+  Returns the latest version of the given npm package.
+
+  Example: [lib/docs/scrapers/bower.rb](../lib/docs/scrapers/bower.rb)
+* `get_latest_github_release(owner, repo, opts)`
+
+  Returns the tag name of the latest GitHub release of the given repository. If the tag name is preceded by a "v", the "v" will be removed.
+
+  Example: [lib/docs/scrapers/jsdoc.rb](../lib/docs/scrapers/jsdoc.rb)
+* `get_github_tags(owner, repo, opts)`
+
+  Returns the list of tags on the given repository ([format](https://developer.github.com/v3/repos/#list-tags)).
+
+  Example: [lib/docs/scrapers/liquid.rb](../lib/docs/scrapers/liquid.rb)
+* `get_github_file_contents(owner, repo, path, opts)`
+
+  Returns the contents of the requested file in the default branch of the given repository.
+
+  Example: [lib/docs/scrapers/minitest.rb](../lib/docs/scrapers/minitest.rb)

lib/docs/core/doc.rb

@@ -152,7 +152,6 @@ def store_meta(store)
       end
     end
 
-
     def initialize
       raise NotImplementedError, "#{self.class} is an abstract class and cannot be instantiated." if self.class.abstract
     end
@@ -164,5 +163,104 @@ def build_page(id, &block)
     def build_pages(&block)
       raise NotImplementedError
     end
+
+    def get_scraper_version(opts)
+      if self.class.method_defined?(:options) and !options[:release].nil?
+        options[:release]
+      else
+        # If options[:release] does not exist, we return the Epoch timestamp of when the doc was last modified in DevDocs production
+        json = fetch_json('https://devdocs.io/docs.json', opts)
+        items = json.select {|item| item['name'] == self.class.name}
+        items = items.map {|item| item['mtime']}
+        items.max
+      end
+    end
+
+    # Should return the latest version of this documentation
+    # If options[:release] is defined, it should be in the same format
+    # If options[:release] is not defined, it should return the Epoch timestamp of when the documentation was last updated
+    # If the docs will never change, simply return '1.0.0'
+    def get_latest_version(opts)
+      raise NotImplementedError
+    end
+
+    # Returns whether or not this scraper is outdated.
+    #
+    # The default implementation assumes the documentation uses a semver(-like) approach when it comes to versions.
+    # Patch updates are ignored because there are usually little to no documentation changes in bug-fix-only releases.
+    #
+    # Scrapers of documentations that do not use this versioning approach should override this method.
+    #
+    # Examples of the default implementation:
+    # 1 -> 2 = outdated
+    # 1.1 -> 1.2 = outdated
+    # 1.1.1 -> 1.1.2 = not outdated
+    def is_outdated(scraper_version, latest_version)
+      scraper_parts = scraper_version.to_s.split(/\./).map(&:to_i)
+      latest_parts = latest_version.to_s.split(/\./).map(&:to_i)
+
+      # Only check the first two parts, the third part is for patch updates
+      [0, 1].each do |i|
+        break if i >= scraper_parts.length or i >= latest_parts.length
+        return true if latest_parts[i] > scraper_parts[i]
+        return false if latest_parts[i] < scraper_parts[i]
+      end
+
+      false
+    end
+
+    private
+
+    #
+    # Utility methods for get_latest_version
+    #
+
+    def fetch(url, opts)
+      headers = {}
+
+      if opts.key?(:github_token) and url.start_with?('https://api.github.com/')
+        headers['Authorization'] = "token #{opts[:github_token]}"
+      end
+
+      opts[:logger].debug("Fetching #{url}")
+      response = Request.run(url, { connecttimeout: 15, headers: headers })
+
+      if response.success?
+        response.body
+      else
+        reason = response.timed_out? ? "Timed out while connecting to #{url}" : "Couldn't fetch #{url} (response code #{response.code})"
+        opts[:logger].error(reason)
+        raise reason
+      end
+    end
+
+    def fetch_doc(url, opts)
+      body = fetch(url, opts)
+      Nokogiri::HTML.parse(body, nil, 'UTF-8')
+    end
+
+    def fetch_json(url, opts)
+      JSON.parse fetch(url, opts)
+    end
+
+    def get_npm_version(package, opts)
+      json = fetch_json("https://registry.npmjs.com/#{package}", opts)
+      json['dist-tags']['latest']
+    end
+
+    def get_latest_github_release(owner, repo, opts)
+      release = fetch_json("https://api.github.com/repos/#{owner}/#{repo}/releases/latest", opts)
+      tag_name = release['tag_name']
+      tag_name.start_with?('v') ? tag_name[1..-1] : tag_name
+    end
+
+    def get_github_tags(owner, repo, opts)
+      fetch_json("https://api.github.com/repos/#{owner}/#{repo}/tags", opts)
+    end
+
+    def get_github_file_contents(owner, repo, path, opts)
+      json = fetch_json("https://api.github.com/repos/#{owner}/#{repo}/contents/#{path}", opts)
+      Base64.decode64(json['content'])
+    end
   end
 end

lib/docs/scrapers/angular.rb

@@ -155,6 +155,10 @@ def handle_response(response)
       end
     end
 
+    def get_latest_version(opts)
+      get_npm_version('@angular/core', opts)
+    end
+
     private
 
     def parse(response)

lib/docs/scrapers/angularjs.rb

@@ -69,5 +69,9 @@ class Angularjs < UrlScraper
       self.release = '1.2.32'
       self.base_url = "https://code.angularjs.org/#{release}/docs/partials/"
     end
+
+    def get_latest_version(opts)
+      get_npm_version('angular', opts)
+    end
   end
 end

lib/docs/scrapers/ansible.rb

@@ -87,5 +87,10 @@ class Ansible < UrlScraper
         quickstart.html
         list_of_all_modules.html)
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://docs.ansible.com/ansible/latest/index.html', opts)
+      doc.at_css('.DocSiteProduct-CurrentVersion').content.strip
+    end
   end
 end