This file is an attempt to start to document how the reporter.apache.org site is set up.

It is currently very rudimentary.

Original version:
  Javascript and CSS are Foundation
  http://foundation.zurb.com
  Current version seems to be 5.5.1 (see start of site/css/foundation.css)

The site runs on the host projects-vm3.a.o at present (May 2021)

The puppet config for the VM is stored at:
https://github.com/apache/infrastructure-p6/blob/production/data/nodes/projects-vm3.apache.org.yaml
(private repo, requires GH account linked to ASF)

There are other puppet data files for the host under:
https://github.com/apache/infrastructure-p6/blob/production/modules/projects_pvm_asf/

The ASF-hosted private Git repo is at:
https://gitbox.apache.org/repos/private/infrastructure?p=infrastructure-p6.git

Crontab:

crontab -l -u root (in puppet, part of https://github.com/apache/infrastructure-p6/blob/production/modules/projects_pvm_asf/manifests/init.pp):
# m h  dom mon dow   command

# Puppet Name: rao_releases_ci
45 0,6,12,18 * * * cd /var/www/reporter.apache.org/data/releases && sudo -n -u www-data svn ci -m "updating report releases data" --username projects_role --password `cat /root/.rolepwd` --non-interactive >>/var/log/www-data-root/svnreleases_$(date "+\%Y-\%m").log
# Puppet Name: rao_history_ci
10 4,12,20 * * * cd /var/www/reporter.apache.org/data/history && sudo -n -u www-data svn ci -m "updating report releases data" --username projects_role --password `cat /root/.rolepwd` --non-interactive >>/var/log/www-data-root/svnhistory_$(date "+\%Y-\%m").log

crontab -l -u www-data:
# m h   dom mon dow   command
00 4,12,20 * * * cd /var/www/reporter.apache.org/scripts && ./python3logger.sh parsepmcs.py
00 01 * * * cd /var/www/reporter.apache.org/scripts && ./python3logger.sh mailglomper2.py
00 09 * * *      cd /var/www/reporter.apache.org/scripts && ./python3logger.sh readjira.py
10 00 * * *      cd /var/www/reporter.apache.org/scripts && ./python3logger.sh reportingcycles.py
20 00 * * *      cd /var/www/reporter.apache.org/scripts && ./python3logger.sh pmcdates.py
30 00 * * *      cd /var/www/reporter.apache.org/scripts && ./python3logger.sh bugzillastats.py
50 00 * * *      cd /var/www/reporter.apache.org/scripts && ./python3logger.sh health.py

# ensure that any new data files get picked up by the commit (which must be done by root)
40  * * * *      cd /var/www/reporter.apache.org/scripts && ./svnadd.sh ../data/releases

# Run scandist
@reboot         cd /var/www/reporter.apache.org/scripts && ./scandist.sh
@monthly        cd /var/www/reporter.apache.org/scripts && ./scandist.sh restart

The Reporter API uses gunicorn and is now installed in Puppet as a service

There are additional jobs for projects.a.o which are documented in its file STRUCTURE.txt

The application has 3 parts:
- scripts - for crontab or local use
- site - website
- data - long-term data held in SVN as well as URL cache and other ephemeral files

All variable data should now be under the data/ tree.
This means that only the data/ tree needs to be writable by the webserver.

Scripts:
- scripts/health.py
  Creates data/health.json
  Reads:
  data/maildata_extended.json
  data/pmcs.json
  data/projects.json
  data/releases/%s.json % project
  https://whimsy.apache.org/public/committee-info.json via committee_info

- scripts/parsepmcs.py
  Updates data/pmcs.json and data/projects.json (from Whimsy public data)
  Also updates historic copies (without the last seen timestamp) in data/history

-scripts/pmcdates.py
  Creates data/pmcdates.json from committee_info.json

-scripts/reportingcycles.py
  Creates data/reportingcycles.json from committee_info.json

- scripts/mailglomper2.py
  Updates data/maildata_extended.json and data/maildata_partial.json from https://lists.apache.org/

- scripts/readjira.py
   For each .json file found under data/JSON (apart from jira_projects.json) it recreates the file.
   Also refreshes data/JIRA/jira_projects.json

- scripts/bugzillastats.py
  Creates data/bugzillastats.json

- site/addrelease.py
  Updates data/releases/%s.json % committee from form data

Data file consumers:
- chi.py
  data/health.json

- scripts/pdata.py
  data/JIRA/jira_projects.json
  data/JIRA/%s.json
  data/health.json
  data/releases/%s.json
  data/pmcs.json
  data/projects.json
  data/maildata_extended.json
  https://whimsy.apache.org/public/public_ldap_people.json
  https://whimsy.apache.org/public/member-info.json
  https://whimsy.apache.org/public/public_ldap_projects.json
  https://projects.apache.org/json/foundation/committees.json