Layout of comdev/projects.apache.org: /scripts: - Contains scripts used for import and maintenance of foundation-wide data, such as committer IDs/names, project VPs, founding dates, reporting cycles etc. See /scripts/README.txt for details. /scripts/cronjobs: - Cronjob scripts that run nightly on projects-vm3 to generate JSON from authoritative sources (Whimsy, LDAP, Incubator, dist/). Scripts: parsecommitters.py - committer/group data from Whimsy LDAP parsecommitteeinfo.py - PMC data from Whimsy committee-info.json + local RDF podlings.py - podling data from Incubator podlings.xml parsereleases.py - release data from dist.apache.org parseprojects.py - project metadata from DOAP files generaterepos.py - repository listing pubsubber.py - listens for SVN commits (runs on reboot) Utilities: python3logger.sh - wrapper that logs script output svnadd.sh - SVN-adds new JSON files before nightly commit jsonutils.py - shared JSON helpers urlutils.py - shared URL fetch helpers errtee.py - error logging helper sendmail.py - email notification helper testlogging.py - logging test /data: - Contains data maintained by committees /data/committees.xml: - List of where to find PMC description RDF files /data/committees/: - Per-committee RDF description files (*.rdf) /data/committees-retired/: - RDF files for retired committees /data/projects.xml: - Master list of DOAP file locations for all projects /data/projects-override/: - Local copies of DOAP files for retired projects (moved to Attic). Used when the original project committers are unavailable to update the DOAP themselves. See AAREADME.txt in that directory for the update procedure. /data/cache/: - Cached data from external sources /failures/: - (currently empty) Placeholder for failed cronjob output /site: - Contains the HTML, images and javascript needed to run the site /site/index.html, committees.html, project.html, projects.html, etc.: - Static HTML pages that load JSON data client-side via JavaScript /site/js/: - JavaScript: projects.js (main logic), script.js, project_editor.js, plus libraries (jQuery, DataTables, ECharts, Underscore) /site/css/: - Stylesheets /site/images/: - Images /site/indexes/: - Alternate project listing views: alphabetical, by category, by language, by PMC, releases, standards, quick search /site/edit/: - DISABLED. Was an LDAP-authenticated DOAP editor (CGI-based). See README.txt in that directory. /site/ns/: - ASF extension namespace definition (asfext) /site/json: - Contains the JSON data storage calculated from /data and external sources. Also used by reporter.apache.org (see getjson.py). /site/json/foundation: - Foundation-wide JSON data generated by cronjobs: committees.json - PMC roster, chairs, charters committees-retired.json - retired committees people.json - committers with group membership people_name.json - availid → public name mapping groups.json - groups with their committers podlings.json - current podlings podlings-history.json - ended/graduated podlings releases.json - release data per project releases-files.json - individual release files projects.json - combined listing from DOAP files repositories.json - Git/SVN repository listing accounts-evolution2.json - account creation timeline /site/json/projects: - Per-project JSON files parsed from DOAP files by parseprojects.py. One file per project (e.g. kafka.json, httpd.json). /Docker support: Dockerfile, compose.yml, DOCKER.md - local development setup. docker-config/ - Apache httpd config (25-projects.conf) + setup script docker-data/ - runtime data (logs/) for Docker containers Note: the directory structure on the VM should be owned by the www-data user (or whatever is used for the webserver), because at least one script (parsecommitteeinfo.py) may invoke SVN commands. Webserver required: To test the site locally without Docker, a webserver is required or you'll get "Cross origin requests are only supported for HTTP" errors. An easy setup for development is: run "python3 -m http.server 8888" from the site directory to have the site available at http://localhost:8888/ Current crontab settings (projects-vm3): crontab root: # m h dom mon dow command 20 4 * * * cd /var/www/projects.apache.org/site/json && sudo -n -u www-data svn ci -m "updating projects data" --username projects_role --password `cat /root/.rolepwd` --non-interactive crontab -l -u www-data: # m h dom mon dow command 00 00 * * * cd /var/www/projects.apache.org/scripts/cronjobs && ./python3logger.sh podlings.py 01 00 * * * cd /var/www/projects.apache.org/scripts/cronjobs && ./python3logger.sh parsecommitters.py 03 00 * * * cd /var/www/projects.apache.org/scripts/cronjobs && ./python3logger.sh parsereleases.py 05 01 * * * cd /var/www/projects.apache.org/scripts/cronjobs && ./python3logger.sh parsecommitteeinfo.py 00 02 * * * cd /var/www/projects.apache.org/scripts/cronjobs && ./python3logger.sh parseprojects.py 00 03 * * * cd /var/www/projects.apache.org/scripts/cronjobs && ./python3logger.sh generaterepos.py # Run pubsubber @reboot cd /var/www/projects.apache.org/scripts/cronjobs && ./pubsubber.sh @monthly cd /var/www/projects.apache.org/scripts/cronjobs && ./pubsubber.sh restart # ensure that any new data files get picked up by the commit (which must be started by root) 10 4 * * * cd /var/www/projects.apache.org/scripts/cronjobs && ./svnadd.sh ../../site/json There are additional jobs for reporter.a.o which are documented in its code source (README.txt). Note: the puppet config for the VM is stored at: https://github.com/apache/infrastructure-p6/blob/production/data/nodes/projects-vm3.apache.org.yaml and https://github.com/apache/infrastructure-p6/tree/production/modules/projects_pvm_asf See also scripts/README.txt