Examine individual changes
From Gentoo Wiki
This page allows you to examine the variables generated by the Abuse Filter for an individual change, and test it against filters.
Variables generated for this change
| Variable | Value |
|---|---|
Edit count of the user (user_editcount) | 3175 |
Name of the user account (user_name) | 'Sam' |
Age of the user account (user_age) | 207907885 |
Page ID (page_id) | 197366 |
Page namespace (page_namespace) | 510 |
Page title (without namespace) (page_title) | 'Gentoostats' |
Full page title (page_prefixedtitle) | 'Project:Gentoostats' |
Action (action) | 'edit' |
Edit summary/reason (summary) | 'Retire dolsen' |
Old content model (old_content_model) | 'wikitext' |
New content model (new_content_model) | 'wikitext' |
Old page wikitext, before the edit (old_wikitext) | '{{Project
|Name=Gentoostats
|Description=Gentoostats project maintains and develops the "gentoostats" statistics collection software for Gentoo machines
|Email=gentoostats@gentoo.org
|Bug assignment=No
|Packages=No
|ParentProject=Project:Gentoo
|PropagatesMembers=No
|LeadElectionDate=2017-01-02
|Members={{Project Member
|Developer=User:Gokturk
|Role=Lead
|IsLead=Yes
}}{{Project Member
|Developer=User:Dilfridge
|Role=Moral support
|IsLead=No
}}{{Project Member
|Developer=User:Dol-sen
|Role=Portage API
|IsLead=No
}}
}}
Gentoostats project tasks itself with the [[Article description::deployment, maintenance, and continued development of {{c|gentoostats}}]], a software that collects various statistics from Gentoo machines.
== About Gentoostats ==
Gentoostats is written by Vikraman Choudhury as a Google Summer Code 2011 [https://www.google-melange.com/archive/gsoc/2011/orgs/gentoo/projects/vh4x0r.html project]. It is written in Python and implements a client-server model. The server component is a [https://www.python.org/dev/peps/pep-0333/ WSGI] web application, built using the [http://webpy.org/ webpy] framework. The client component uses the Portage API to collect various statistics from a Gentoo machine, encodes it in the [https://tools.ietf.org/html/rfc7159 JSON format] and submits it to the server. Users have the ability to configure which information is to be transmitted according to their privacy needs.
{{WIP|author=gokturk}}
== Suggested features ==
=== Upload package build time statistics (veremit) ===
Instead of reporting it in absolute time, look into using a relative measure like SBU (see:[[wikipedia:Linux_From_Scratch#Standard_build_unit|SBU]]).
=== Distributed collection servers ===
Make multiple servers exchange and sync stats with each other. This is similar to the pgp keyservers and the goal is to distribute the load and combat DDOS. The major problem is the collision of host UUIDs across multiple independent servers. Second problem is the trust between servers for which solutions exist. Since the database will grow significantly large, some form of delta-sync will be necessary.
== Open problems ==
=== Validity of the submitted samples ===
There's no mechanism to stop a malicious user from flooding the server by synthetically creating a large set of statistics reports and submitting them. There's no way to prove that the submitted statistics come from an actual installation. This can be utilized in the form of denial of service or skewing the statistics. Some form of rate limiting and snapshotting may be useful.
== TODO ==
* Make an initial release of gentoostats server and add it to the tree
* Work with infra about gentoostats deployment
* Update [[Gentoostats]], add a section for deploying private instances, improve the usage text
=== Discussion regarding which version to deploy ===
{| class="table table-striped table-condensed"
|-
! !! Gentoostats 2011 !! Gentoostats 2012
|-
| Pros ||
* Simplistic design based on web.py with small number of dependencies
* With near to no maintenance over 5 years, it was still almost completely functional
* Submitting stats is much faster compared to Gentoostats 2012
||
* Based on django and has potential for richer web interfaces
* Use of django models provides a good data abstraction and makes the solution independent of a particular SQLDB
|-
| Cons ||
* Archaic web interface
* Directly deals with SQL queries over web.py, no abstraction
* No python3 support for web.py yet
||
* Maintenance becomes a burden. Need to keep up with django upgrades.
* Submitting statistics works way more slowly
* Increased code complexity
|}
=== Gentoostats 2011 ===
* Code: https://gitweb.gentoo.org/proj/gentoostats.git/
Progress reports:
* Progress Report #1: https://archives.gentoo.org/gentoo-soc/message/2f9044ad5390b53a338fc9bca4bebda5
* Progress Report #2: https://archives.gentoo.org/gentoo-soc/message/b345988ca5df929abb4f0f5b9aceb00c
* Progress Report #3: https://archives.gentoo.org/gentoo-soc/message/845d373851b7b06f4ab7ce3662b15b4b
* Progress Report #4: https://archives.gentoo.org/gentoo-soc/message/76a0eb1b38e9101ca44d5da7723dcf60
* Progress Report #5: https://archives.gentoo.org/gentoo-soc/message/0179caaa96f8df9f4619a38d630c8cdb
* Midterm Report: https://archives.gentoo.org/gentoo-soc/message/a982111423d18fb7a714526bf9052708
* Progress Report #6: https://archives.gentoo.org/gentoo-soc/message/635ee0e2c9e3d599be5e9c05cd905f9c
* Progress Report #7: https://archives.gentoo.org/gentoo-soc/message/606094a198354a2938b8b8a10f7b0cb5
* Final Report: https://archives.gentoo.org/gentoo-soc/message/c90536fdd571898e6a15c6c7d9fa0c75
=== Gentoostats 2012 ===
Apparently, there's another gentoostats project based on django written as part of GSoC 2012:
* Server: https://github.com/gg7/gentoostats_server
* Client: https://github.com/gg7/gentoostats
* Playground (??): https://github.com/vikraman/gentoostats-playground
* Deployment bug: https://bugs.gentoo.org/show_bug.cgi?id=425056
Progress reports:
* Progress Report #1: https://archives.gentoo.org/gentoo-soc/message/a85db0776186d6e4fa032377af2c8634
* Progress Report #2: https://archives.gentoo.org/gentoo-soc/message/b0be0d2f6a5c43457ef6cebd3f8e9b7b
* Progress Report #3: https://archives.gentoo.org/gentoo-soc/message/1b45015692cecc31211f93de4bb701d0
* Progress Report #4: https://archives.gentoo.org/gentoo-soc/message/1e1a675494bca49352097a0b25dd58f9
* Progress Report #5: https://archives.gentoo.org/gentoo-soc/message/a8a0f843bd2b755f834b3f9eacdbf97b
* Progress Report #6: https://archives.gentoo.org/gentoo-soc/message/e8a9ef1386d0bf29d922a86ee5332ea8
* Progress Report #7: https://archives.gentoo.org/gentoo-soc/message/8e9fcbd3ab67cdc7c66c9aab87eea62f
* Final Report: https://archives.gentoo.org/gentoo-soc/message/760cbd58a309b56f31d3697d90f44601
Find out why the code isn't being hosted on infra. Evaluate the functionality. Determine which version is to be deployed and maintained.
==== Attempting to deploy Gentoostats 2012 ====
This is an ongoing effort to deploy this version of gentoostats on my local machine:
Package list:
* dev-python/django-1.8.9
* dev-python/django-extensions-1.6.1
* dev-python/django-debug-toolbar-1.3.2
* dev-python/django-tastypie-0.9.15
Steps:
* Clone the repo, copy gentoostats/settings.py.example to gentoostats/settings.py and edit accordingly
* "manage.py check" dies with "ImportError: No module named south"
** south is hard masked
** comment out south from INSTALLED_APPS in settings.py
* "manage.py" check dies with the following:
File "/usr/lib64/python2.7/site-packages/tastypie/resources.py", line 2256, in ModelResource @transaction.commit_on_success()
AttributeError: 'module' object has no attribute 'commit_on_success'
As a hackaroo, edit tastypie and replace "@transaction.commit_on_success" with "@transaction.atomic", see: https://github.com/macropin/django-registration/issues/51#issuecomment-100579391
Do the same in gentoostats/receivers/views.py.
* Initialize the database with "manage.py syncdb"
/usr/lib64/python2.7/site-packages/django/core/management/commands/syncdb.py:24: RemovedInDjango19Warning: The syncdb command will be removed in Django 1.9
* Run the server with "manage.py runserver"
January 03, 2017 - 01:01:37
Django version 1.8.9, using settings 'gentoostats.settings'
Starting development server at http://127.0.0.1:8000/
Dies with "ImportError: No module named transaction"
** Comment out 'django.middleware.transaction.TransactionMiddleware' from MIDDLEWARE_CLASSES in settings.py, see: http://stackoverflow.com/a/33102743
* Try to upload stats, dies with:
INFO 2017-01-03 01:50:31,928 views 27822 140719396648704 process_submission(): Error: Invalid date in LASTSYNC.
Traceback (most recent call last):
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 88, in process_submission
time.strptime(lastsync, "%a, %d %b %Y %H:%M:%S +0000")
File "/usr/lib64/python2.7/_strptime.py", line 478, in _strptime_time
return _strptime(data_string, format)[0]
File "/usr/lib64/python2.7/_strptime.py", line 332, in _strptime
(data_string, format))
ValueError: time data u'Unknown' does not match format '%a, %d %b %Y %H:%M:%S +0000'
This is due to using a git repo in /usr/portage, which doesn't contain the timestamp file and the client is sending the string "Unknown" after patching with similarly to https://gitweb.gentoo.org/proj/gentoostats.git/commit/?id=963afe1163125b8cbed08c0e8edea9a05a37510e. Patch it with
- if lastsync:
+ if lastsync and lastsync != "Unknown":
and add the following else statement to it:
else:
lastsync = None
* Try to upload stats again, dies with:
ERROR 2017-01-03 02:02:24,104 views 810 140115441485568 process_submission(): 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 369, in accept_submission
return process_submission(request)
File "/usr/lib64/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/django/utils/decorators.py", line 145, in inner
return func(*args, **kwargs)
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 159, in process_submission
country = GeoIP().country_name(ip_addr),
File "/usr/lib64/python2.7/site-packages/django/contrib/gis/geoip/base.py", line 190, in country_name
return self.city(query)['country_name']
TypeError: 'NoneType' object has no attribute '__getitem__'
Comment out the call to "GeoIP().country_name(ip_addr)" in gentoostats/receiver/view.py for now.
* Try to upload stats again, dies with:
ERROR 2017-01-03 02:26:40,646 views 13386 140066905986816 process_submission(): Cannot assign "u''": "Submission.sync" must be a "SyncServer" instance.
Traceback (most recent call last):
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 369, in accept_submission
return process_submission(request)
File "/usr/lib64/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/django/utils/decorators.py", line 145, in inner
return func(*args, **kwargs)
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 186, in process_submission
sync = sync,
File "/usr/lib64/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/django/db/models/query.py", line 346, in create
obj = self.model(**kwargs)
File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", line 468, in __init__
setattr(self, field.name, rel_obj)
File "/usr/lib64/python2.7/site-packages/django/db/models/fields/related.py", line 642, in __set__
self.field.rel.to._meta.object_name,
ValueError: Cannot assign "u''": "Submission.sync" must be a "SyncServer" instance.
This is due to gentoostats expecting the SYNC variable in make.conf. From gentoostats/stats/models.py:
# make.conf example: SYNC="rsync://rsync.gentoo.org/gentoo-portage"
sync = models.ForeignKey(SyncServer, blank=True, null=True, related_name='+')
Set sync to None for now:
@@ -145,6 +147,7 @@ def process_submission(request):
validate_item(lang)
sync = data.get('SYNC')
+ sync = None
if sync:
sync, _ = SyncServer.objects.get_or_create(url=sync)
validate_item(sync)
* Try to upload stats again, viola!' |
New page wikitext, after the edit (new_wikitext) | '{{Project
|Name=Gentoostats
|Description=Gentoostats project maintains and develops the "gentoostats" statistics collection software for Gentoo machines
|Email=gentoostats@gentoo.org
|Bug assignment=No
|Packages=No
|ParentProject=Project:Gentoo
|PropagatesMembers=No
|LeadElectionDate=2017-01-02
|Members={{Project Member
|Developer=User:Gokturk
|Role=Lead
|IsLead=Yes
}}{{Project Member
|Developer=User:Dilfridge
|Role=Moral support
|IsLead=No
}}
}}
Gentoostats project tasks itself with the [[Article description::deployment, maintenance, and continued development of {{c|gentoostats}}]], a software that collects various statistics from Gentoo machines.
== About Gentoostats ==
Gentoostats is written by Vikraman Choudhury as a Google Summer Code 2011 [https://www.google-melange.com/archive/gsoc/2011/orgs/gentoo/projects/vh4x0r.html project]. It is written in Python and implements a client-server model. The server component is a [https://www.python.org/dev/peps/pep-0333/ WSGI] web application, built using the [http://webpy.org/ webpy] framework. The client component uses the Portage API to collect various statistics from a Gentoo machine, encodes it in the [https://tools.ietf.org/html/rfc7159 JSON format] and submits it to the server. Users have the ability to configure which information is to be transmitted according to their privacy needs.
{{WIP|author=gokturk}}
== Suggested features ==
=== Upload package build time statistics (veremit) ===
Instead of reporting it in absolute time, look into using a relative measure like SBU (see:[[wikipedia:Linux_From_Scratch#Standard_build_unit|SBU]]).
=== Distributed collection servers ===
Make multiple servers exchange and sync stats with each other. This is similar to the pgp keyservers and the goal is to distribute the load and combat DDOS. The major problem is the collision of host UUIDs across multiple independent servers. Second problem is the trust between servers for which solutions exist. Since the database will grow significantly large, some form of delta-sync will be necessary.
== Open problems ==
=== Validity of the submitted samples ===
There's no mechanism to stop a malicious user from flooding the server by synthetically creating a large set of statistics reports and submitting them. There's no way to prove that the submitted statistics come from an actual installation. This can be utilized in the form of denial of service or skewing the statistics. Some form of rate limiting and snapshotting may be useful.
== TODO ==
* Make an initial release of gentoostats server and add it to the tree
* Work with infra about gentoostats deployment
* Update [[Gentoostats]], add a section for deploying private instances, improve the usage text
=== Discussion regarding which version to deploy ===
{| class="table table-striped table-condensed"
|-
! !! Gentoostats 2011 !! Gentoostats 2012
|-
| Pros ||
* Simplistic design based on web.py with small number of dependencies
* With near to no maintenance over 5 years, it was still almost completely functional
* Submitting stats is much faster compared to Gentoostats 2012
||
* Based on django and has potential for richer web interfaces
* Use of django models provides a good data abstraction and makes the solution independent of a particular SQLDB
|-
| Cons ||
* Archaic web interface
* Directly deals with SQL queries over web.py, no abstraction
* No python3 support for web.py yet
||
* Maintenance becomes a burden. Need to keep up with django upgrades.
* Submitting statistics works way more slowly
* Increased code complexity
|}
=== Gentoostats 2011 ===
* Code: https://gitweb.gentoo.org/proj/gentoostats.git/
Progress reports:
* Progress Report #1: https://archives.gentoo.org/gentoo-soc/message/2f9044ad5390b53a338fc9bca4bebda5
* Progress Report #2: https://archives.gentoo.org/gentoo-soc/message/b345988ca5df929abb4f0f5b9aceb00c
* Progress Report #3: https://archives.gentoo.org/gentoo-soc/message/845d373851b7b06f4ab7ce3662b15b4b
* Progress Report #4: https://archives.gentoo.org/gentoo-soc/message/76a0eb1b38e9101ca44d5da7723dcf60
* Progress Report #5: https://archives.gentoo.org/gentoo-soc/message/0179caaa96f8df9f4619a38d630c8cdb
* Midterm Report: https://archives.gentoo.org/gentoo-soc/message/a982111423d18fb7a714526bf9052708
* Progress Report #6: https://archives.gentoo.org/gentoo-soc/message/635ee0e2c9e3d599be5e9c05cd905f9c
* Progress Report #7: https://archives.gentoo.org/gentoo-soc/message/606094a198354a2938b8b8a10f7b0cb5
* Final Report: https://archives.gentoo.org/gentoo-soc/message/c90536fdd571898e6a15c6c7d9fa0c75
=== Gentoostats 2012 ===
Apparently, there's another gentoostats project based on django written as part of GSoC 2012:
* Server: https://github.com/gg7/gentoostats_server
* Client: https://github.com/gg7/gentoostats
* Playground (??): https://github.com/vikraman/gentoostats-playground
* Deployment bug: https://bugs.gentoo.org/show_bug.cgi?id=425056
Progress reports:
* Progress Report #1: https://archives.gentoo.org/gentoo-soc/message/a85db0776186d6e4fa032377af2c8634
* Progress Report #2: https://archives.gentoo.org/gentoo-soc/message/b0be0d2f6a5c43457ef6cebd3f8e9b7b
* Progress Report #3: https://archives.gentoo.org/gentoo-soc/message/1b45015692cecc31211f93de4bb701d0
* Progress Report #4: https://archives.gentoo.org/gentoo-soc/message/1e1a675494bca49352097a0b25dd58f9
* Progress Report #5: https://archives.gentoo.org/gentoo-soc/message/a8a0f843bd2b755f834b3f9eacdbf97b
* Progress Report #6: https://archives.gentoo.org/gentoo-soc/message/e8a9ef1386d0bf29d922a86ee5332ea8
* Progress Report #7: https://archives.gentoo.org/gentoo-soc/message/8e9fcbd3ab67cdc7c66c9aab87eea62f
* Final Report: https://archives.gentoo.org/gentoo-soc/message/760cbd58a309b56f31d3697d90f44601
Find out why the code isn't being hosted on infra. Evaluate the functionality. Determine which version is to be deployed and maintained.
==== Attempting to deploy Gentoostats 2012 ====
This is an ongoing effort to deploy this version of gentoostats on my local machine:
Package list:
* dev-python/django-1.8.9
* dev-python/django-extensions-1.6.1
* dev-python/django-debug-toolbar-1.3.2
* dev-python/django-tastypie-0.9.15
Steps:
* Clone the repo, copy gentoostats/settings.py.example to gentoostats/settings.py and edit accordingly
* "manage.py check" dies with "ImportError: No module named south"
** south is hard masked
** comment out south from INSTALLED_APPS in settings.py
* "manage.py" check dies with the following:
File "/usr/lib64/python2.7/site-packages/tastypie/resources.py", line 2256, in ModelResource @transaction.commit_on_success()
AttributeError: 'module' object has no attribute 'commit_on_success'
As a hackaroo, edit tastypie and replace "@transaction.commit_on_success" with "@transaction.atomic", see: https://github.com/macropin/django-registration/issues/51#issuecomment-100579391
Do the same in gentoostats/receivers/views.py.
* Initialize the database with "manage.py syncdb"
/usr/lib64/python2.7/site-packages/django/core/management/commands/syncdb.py:24: RemovedInDjango19Warning: The syncdb command will be removed in Django 1.9
* Run the server with "manage.py runserver"
January 03, 2017 - 01:01:37
Django version 1.8.9, using settings 'gentoostats.settings'
Starting development server at http://127.0.0.1:8000/
Dies with "ImportError: No module named transaction"
** Comment out 'django.middleware.transaction.TransactionMiddleware' from MIDDLEWARE_CLASSES in settings.py, see: http://stackoverflow.com/a/33102743
* Try to upload stats, dies with:
INFO 2017-01-03 01:50:31,928 views 27822 140719396648704 process_submission(): Error: Invalid date in LASTSYNC.
Traceback (most recent call last):
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 88, in process_submission
time.strptime(lastsync, "%a, %d %b %Y %H:%M:%S +0000")
File "/usr/lib64/python2.7/_strptime.py", line 478, in _strptime_time
return _strptime(data_string, format)[0]
File "/usr/lib64/python2.7/_strptime.py", line 332, in _strptime
(data_string, format))
ValueError: time data u'Unknown' does not match format '%a, %d %b %Y %H:%M:%S +0000'
This is due to using a git repo in /usr/portage, which doesn't contain the timestamp file and the client is sending the string "Unknown" after patching with similarly to https://gitweb.gentoo.org/proj/gentoostats.git/commit/?id=963afe1163125b8cbed08c0e8edea9a05a37510e. Patch it with
- if lastsync:
+ if lastsync and lastsync != "Unknown":
and add the following else statement to it:
else:
lastsync = None
* Try to upload stats again, dies with:
ERROR 2017-01-03 02:02:24,104 views 810 140115441485568 process_submission(): 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 369, in accept_submission
return process_submission(request)
File "/usr/lib64/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/django/utils/decorators.py", line 145, in inner
return func(*args, **kwargs)
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 159, in process_submission
country = GeoIP().country_name(ip_addr),
File "/usr/lib64/python2.7/site-packages/django/contrib/gis/geoip/base.py", line 190, in country_name
return self.city(query)['country_name']
TypeError: 'NoneType' object has no attribute '__getitem__'
Comment out the call to "GeoIP().country_name(ip_addr)" in gentoostats/receiver/view.py for now.
* Try to upload stats again, dies with:
ERROR 2017-01-03 02:26:40,646 views 13386 140066905986816 process_submission(): Cannot assign "u''": "Submission.sync" must be a "SyncServer" instance.
Traceback (most recent call last):
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 369, in accept_submission
return process_submission(request)
File "/usr/lib64/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/django/utils/decorators.py", line 145, in inner
return func(*args, **kwargs)
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 186, in process_submission
sync = sync,
File "/usr/lib64/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/django/db/models/query.py", line 346, in create
obj = self.model(**kwargs)
File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", line 468, in __init__
setattr(self, field.name, rel_obj)
File "/usr/lib64/python2.7/site-packages/django/db/models/fields/related.py", line 642, in __set__
self.field.rel.to._meta.object_name,
ValueError: Cannot assign "u''": "Submission.sync" must be a "SyncServer" instance.
This is due to gentoostats expecting the SYNC variable in make.conf. From gentoostats/stats/models.py:
# make.conf example: SYNC="rsync://rsync.gentoo.org/gentoo-portage"
sync = models.ForeignKey(SyncServer, blank=True, null=True, related_name='+')
Set sync to None for now:
@@ -145,6 +147,7 @@ def process_submission(request):
validate_item(lang)
sync = data.get('SYNC')
+ sync = None
if sync:
sync, _ = SyncServer.objects.get_or_create(url=sync)
validate_item(sync)
* Try to upload stats again, viola!' |
Unified diff of changes made by edit (edit_diff) | '@@ -15,8 +15,4 @@
|Developer=User:Dilfridge
|Role=Moral support
-|IsLead=No
-}}{{Project Member
-|Developer=User:Dol-sen
-|Role=Portage API
|IsLead=No
}}
' |
Old page size (old_size) | 11731 |
Lines added in edit (added_lines) | [] |
Lines removed in edit (removed_lines) | [
0 => '|IsLead=No',
1 => '}}{{Project Member',
2 => '|Developer=User:Dol-sen',
3 => '|Role=Portage API'
] |
New page text, stripped of any markup (new_text) | '
Gentoostats
Description
Gentoostats project maintains and develops the "gentoostats" statistics collection software for Gentoo machines
Project email
gentoostats@gentoo.org
Lead(s)
Göktürk Yüksek
(gokturk)Lead Last elected: 2017-01-02
Member(s)
Andreas K. Hüttel
(dilfridge)Moral supportBrian Dolbec
(dolsen)Portage API
Subproject(s)(and inherited member(s))
(none)
Parent Project
Gentoo
Project listing
Gentoostats project tasks itself with the deployment, maintenance, and continued development of gentoostats, a software that collects various statistics from Gentoo machines.
Contents
1 About Gentoostats
2 Suggested features
2.1 Upload package build time statistics (veremit)
2.2 Distributed collection servers
3 Open problems
3.1 Validity of the submitted samples
4 TODO
4.1 Discussion regarding which version to deploy
4.2 Gentoostats 2011
4.3 Gentoostats 2012
4.3.1 Attempting to deploy Gentoostats 2012
About Gentoostats[edit | edit source]
Gentoostats is written by Vikraman Choudhury as a Google Summer Code 2011 project. It is written in Python and implements a client-server model. The server component is a WSGI web application, built using the webpy framework. The client component uses the Portage API to collect various statistics from a Gentoo machine, encodes it in the JSON format and submits it to the server. Users have the ability to configure which information is to be transmitted according to their privacy needs.
Warning, this page is a work in progress by gokturk (talk | contribs). Treat its contents with caution.
Suggested features[edit | edit source]
Upload package build time statistics (veremit)[edit | edit source]
Instead of reporting it in absolute time, look into using a relative measure like SBU (see:SBU).
Distributed collection servers[edit | edit source]
Make multiple servers exchange and sync stats with each other. This is similar to the pgp keyservers and the goal is to distribute the load and combat DDOS. The major problem is the collision of host UUIDs across multiple independent servers. Second problem is the trust between servers for which solutions exist. Since the database will grow significantly large, some form of delta-sync will be necessary.
Open problems[edit | edit source]
Validity of the submitted samples[edit | edit source]
There's no mechanism to stop a malicious user from flooding the server by synthetically creating a large set of statistics reports and submitting them. There's no way to prove that the submitted statistics come from an actual installation. This can be utilized in the form of denial of service or skewing the statistics. Some form of rate limiting and snapshotting may be useful.
TODO[edit | edit source]
Make an initial release of gentoostats server and add it to the tree
Work with infra about gentoostats deployment
Update Gentoostats, add a section for deploying private instances, improve the usage text
Discussion regarding which version to deploy[edit | edit source]
Gentoostats 2011
Gentoostats 2012
Pros
Simplistic design based on web.py with small number of dependencies
With near to no maintenance over 5 years, it was still almost completely functional
Submitting stats is much faster compared to Gentoostats 2012
Based on django and has potential for richer web interfaces
Use of django models provides a good data abstraction and makes the solution independent of a particular SQLDB
Cons
Archaic web interface
Directly deals with SQL queries over web.py, no abstraction
No python3 support for web.py yet
Maintenance becomes a burden. Need to keep up with django upgrades.
Submitting statistics works way more slowly
Increased code complexity
Gentoostats 2011[edit | edit source]
Code: https://gitweb.gentoo.org/proj/gentoostats.git/
Progress reports:
Progress Report #1: https://archives.gentoo.org/gentoo-soc/message/2f9044ad5390b53a338fc9bca4bebda5
Progress Report #2: https://archives.gentoo.org/gentoo-soc/message/b345988ca5df929abb4f0f5b9aceb00c
Progress Report #3: https://archives.gentoo.org/gentoo-soc/message/845d373851b7b06f4ab7ce3662b15b4b
Progress Report #4: https://archives.gentoo.org/gentoo-soc/message/76a0eb1b38e9101ca44d5da7723dcf60
Progress Report #5: https://archives.gentoo.org/gentoo-soc/message/0179caaa96f8df9f4619a38d630c8cdb
Midterm Report: https://archives.gentoo.org/gentoo-soc/message/a982111423d18fb7a714526bf9052708
Progress Report #6: https://archives.gentoo.org/gentoo-soc/message/635ee0e2c9e3d599be5e9c05cd905f9c
Progress Report #7: https://archives.gentoo.org/gentoo-soc/message/606094a198354a2938b8b8a10f7b0cb5
Final Report: https://archives.gentoo.org/gentoo-soc/message/c90536fdd571898e6a15c6c7d9fa0c75
Gentoostats 2012[edit | edit source]
Apparently, there's another gentoostats project based on django written as part of GSoC 2012:
Server: https://github.com/gg7/gentoostats_server
Client: https://github.com/gg7/gentoostats
Playground (??): https://github.com/vikraman/gentoostats-playground
Deployment bug: https://bugs.gentoo.org/show_bug.cgi?id=425056
Progress reports:
Progress Report #1: https://archives.gentoo.org/gentoo-soc/message/a85db0776186d6e4fa032377af2c8634
Progress Report #2: https://archives.gentoo.org/gentoo-soc/message/b0be0d2f6a5c43457ef6cebd3f8e9b7b
Progress Report #3: https://archives.gentoo.org/gentoo-soc/message/1b45015692cecc31211f93de4bb701d0
Progress Report #4: https://archives.gentoo.org/gentoo-soc/message/1e1a675494bca49352097a0b25dd58f9
Progress Report #5: https://archives.gentoo.org/gentoo-soc/message/a8a0f843bd2b755f834b3f9eacdbf97b
Progress Report #6: https://archives.gentoo.org/gentoo-soc/message/e8a9ef1386d0bf29d922a86ee5332ea8
Progress Report #7: https://archives.gentoo.org/gentoo-soc/message/8e9fcbd3ab67cdc7c66c9aab87eea62f
Final Report: https://archives.gentoo.org/gentoo-soc/message/760cbd58a309b56f31d3697d90f44601
Find out why the code isn't being hosted on infra. Evaluate the functionality. Determine which version is to be deployed and maintained.
Attempting to deploy Gentoostats 2012[edit | edit source]
This is an ongoing effort to deploy this version of gentoostats on my local machine:
Package list:
dev-python/django-1.8.9
dev-python/django-extensions-1.6.1
dev-python/django-debug-toolbar-1.3.2
dev-python/django-tastypie-0.9.15
Steps:
Clone the repo, copy gentoostats/settings.py.example to gentoostats/settings.py and edit accordingly
"manage.py check" dies with "ImportError: No module named south"
south is hard masked
comment out south from INSTALLED_APPS in settings.py
"manage.py" check dies with the following:
File "/usr/lib64/python2.7/site-packages/tastypie/resources.py", line 2256, in ModelResource @transaction.commit_on_success()
AttributeError: 'module' object has no attribute 'commit_on_success'
As a hackaroo, edit tastypie and replace "@transaction.commit_on_success" with "@transaction.atomic", see: https://github.com/macropin/django-registration/issues/51#issuecomment-100579391
Do the same in gentoostats/receivers/views.py.
Initialize the database with "manage.py syncdb"
/usr/lib64/python2.7/site-packages/django/core/management/commands/syncdb.py:24: RemovedInDjango19Warning: The syncdb command will be removed in Django 1.9
Run the server with "manage.py runserver"
January 03, 2017 - 01:01:37
Django version 1.8.9, using settings 'gentoostats.settings'
Starting development server at http://127.0.0.1:8000/
Dies with "ImportError: No module named transaction"
Comment out 'django.middleware.transaction.TransactionMiddleware' from MIDDLEWARE_CLASSES in settings.py, see: http://stackoverflow.com/a/33102743
Try to upload stats, dies with:
INFO 2017-01-03 01:50:31,928 views 27822 140719396648704 process_submission(): Error: Invalid date in LASTSYNC.
Traceback (most recent call last):
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 88, in process_submission
time.strptime(lastsync, "%a, %d %b %Y %H:%M:%S +0000")
File "/usr/lib64/python2.7/_strptime.py", line 478, in _strptime_time
return _strptime(data_string, format)[0]
File "/usr/lib64/python2.7/_strptime.py", line 332, in _strptime
(data_string, format))
ValueError: time data u'Unknown' does not match format '%a, %d %b %Y %H:%M:%S +0000'
This is due to using a git repo in /usr/portage, which doesn't contain the timestamp file and the client is sending the string "Unknown" after patching with similarly to https://gitweb.gentoo.org/proj/gentoostats.git/commit/?id=963afe1163125b8cbed08c0e8edea9a05a37510e. Patch it with
- if lastsync:
+ if lastsync and lastsync != "Unknown":
and add the following else statement to it:
else:
lastsync = None
Try to upload stats again, dies with:
ERROR 2017-01-03 02:02:24,104 views 810 140115441485568 process_submission(): 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 369, in accept_submission
return process_submission(request)
File "/usr/lib64/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/django/utils/decorators.py", line 145, in inner
return func(*args, **kwargs)
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 159, in process_submission
country = GeoIP().country_name(ip_addr),
File "/usr/lib64/python2.7/site-packages/django/contrib/gis/geoip/base.py", line 190, in country_name
return self.city(query)['country_name']
TypeError: 'NoneType' object has no attribute '__getitem__'
Comment out the call to "GeoIP().country_name(ip_addr)" in gentoostats/receiver/view.py for now.
Try to upload stats again, dies with:
ERROR 2017-01-03 02:26:40,646 views 13386 140066905986816 process_submission(): Cannot assign "u": "Submission.sync" must be a "SyncServer" instance.
Traceback (most recent call last):
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 369, in accept_submission
return process_submission(request)
File "/usr/lib64/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/django/utils/decorators.py", line 145, in inner
return func(*args, **kwargs)
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 186, in process_submission
sync = sync,
File "/usr/lib64/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/django/db/models/query.py", line 346, in create
obj = self.model(**kwargs)
File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", line 468, in __init__
setattr(self, field.name, rel_obj)
File "/usr/lib64/python2.7/site-packages/django/db/models/fields/related.py", line 642, in __set__
self.field.rel.to._meta.object_name,
ValueError: Cannot assign "u": "Submission.sync" must be a "SyncServer" instance.
This is due to gentoostats expecting the SYNC variable in make.conf. From gentoostats/stats/models.py:
# make.conf example: SYNC="rsync://rsync.gentoo.org/gentoo-portage"
sync = models.ForeignKey(SyncServer, blank=True, null=True, related_name='+')
Set sync to None for now:
@@ -145,6 +147,7 @@ def process_submission(request):
validate_item(lang)
sync = data.get('SYNC')
+ sync = None
if sync:
sync, _ = SyncServer.objects.get_or_create(url=sync)
validate_item(sync)
Try to upload stats again, viola!' |
Parsed HTML source of the new revision (new_html) | '<div class="mw-parser-output"><table class="table table-condensed" style="width: 30em; font-size: 95%; border: 1px solid #ddd; background-color: #f9f9f9; color: black; margin-bottom: 0.5em; margin-left: 1em; padding: 0.2em; float: right; clear: right; text-align:left;">
<tbody><tr>
<th style="text-align: center; background-color:#3E355A; color: white;" colspan="2"><big>Gentoostats</big>
</th></tr>
<tr valign="top">
<th>Description
</th>
<td style="text-align: justify;">Gentoostats project maintains and develops the "gentoostats" statistics collection software for Gentoo machines
</td></tr>
<tr>
<th><span title="Mails to member(s) listed below.">Project email</span>
</th>
<td><a rel="nofollow" class="external text" href="mailto:gentoostats@gentoo.org">gentoostats@gentoo.org</a>
</td></tr>
<tr valign="top">
<th>Lead(s)
</th>
<td><ul><li><a href="/wiki/User:Gokturk" title="User:Gokturk">Göktürk Yüksek</a>
(gokturk)<br /><i>Lead</i></li></ul> <br />Last elected: 2017-01-02
</td></tr>
<tr valign="top">
<th>Member(s)
</th>
<td><ul><li><a href="/wiki/User:Dilfridge" title="User:Dilfridge">Andreas K. Hüttel</a>
(dilfridge)<br /><i>Moral support</i></li><li><a href="/wiki/User:Dol-sen" title="User:Dol-sen">Brian Dolbec</a>
(dolsen)<br /><i>Portage API</i></li></ul>
</td></tr>
<tr valign="top">
<th>Subproject(s)<br /><small style="font-weight: normal;">(and inherited member(s))</small>
</th>
<td>(none)
</td></tr>
<tr>
<th>Parent Project
</th>
<td><a href="/wiki/Project:Gentoo" title="Project:Gentoo">Gentoo</a>
</td></tr>
<tr>
<td colspan="2" style="border-top: 1px solid #ddd; font-size: smaller; text-align: center;"><a href="/wiki/Project:Gentoo" title="Project:Gentoo">Project listing</a>
</td></tr></tbody></table>
<p>Gentoostats project tasks itself with the deployment, maintenance, and continued development of <span style="font-family: monospace; font-size: 95%; font-weight: bold;" class="tripleclick-separator">gentoostats</span>, a software that collects various statistics from Gentoo machines.
</p>
<div id="toc" class="toc" role="navigation" aria-labelledby="mw-toc-heading"><input type="checkbox" role="button" id="toctogglecheckbox" class="toctogglecheckbox" style="display:none" /><div class="toctitle" lang="en" dir="ltr"><h2 id="mw-toc-heading">Contents</h2><span class="toctogglespan"><label class="toctogglelabel" for="toctogglecheckbox"></label></span></div>
<ul>
<li class="toclevel-1 tocsection-1"><a href="#About_Gentoostats"><span class="tocnumber">1</span> <span class="toctext">About Gentoostats</span></a></li>
<li class="toclevel-1 tocsection-2"><a href="#Suggested_features"><span class="tocnumber">2</span> <span class="toctext">Suggested features</span></a>
<ul>
<li class="toclevel-2 tocsection-3"><a href="#Upload_package_build_time_statistics_.28veremit.29"><span class="tocnumber">2.1</span> <span class="toctext">Upload package build time statistics (veremit)</span></a></li>
<li class="toclevel-2 tocsection-4"><a href="#Distributed_collection_servers"><span class="tocnumber">2.2</span> <span class="toctext">Distributed collection servers</span></a></li>
</ul>
</li>
<li class="toclevel-1 tocsection-5"><a href="#Open_problems"><span class="tocnumber">3</span> <span class="toctext">Open problems</span></a>
<ul>
<li class="toclevel-2 tocsection-6"><a href="#Validity_of_the_submitted_samples"><span class="tocnumber">3.1</span> <span class="toctext">Validity of the submitted samples</span></a></li>
</ul>
</li>
<li class="toclevel-1 tocsection-7"><a href="#TODO"><span class="tocnumber">4</span> <span class="toctext">TODO</span></a>
<ul>
<li class="toclevel-2 tocsection-8"><a href="#Discussion_regarding_which_version_to_deploy"><span class="tocnumber">4.1</span> <span class="toctext">Discussion regarding which version to deploy</span></a></li>
<li class="toclevel-2 tocsection-9"><a href="#Gentoostats_2011"><span class="tocnumber">4.2</span> <span class="toctext">Gentoostats 2011</span></a></li>
<li class="toclevel-2 tocsection-10"><a href="#Gentoostats_2012"><span class="tocnumber">4.3</span> <span class="toctext">Gentoostats 2012</span></a>
<ul>
<li class="toclevel-3 tocsection-11"><a href="#Attempting_to_deploy_Gentoostats_2012"><span class="tocnumber">4.3.1</span> <span class="toctext">Attempting to deploy Gentoostats 2012</span></a></li>
</ul>
</li>
</ul>
</li>
</ul>
</div>
<h2><span class="mw-headline" id="About_Gentoostats">About Gentoostats</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/index.php?title=Project:Gentoostats&veaction=edit&section=1" class="mw-editsection-visualeditor" title="Edit section: About Gentoostats">edit</a><span class="mw-editsection-divider"> | </span><a href="/index.php?title=Project:Gentoostats&action=edit&section=1" title="Edit section: About Gentoostats">edit source</a><span class="mw-editsection-bracket">]</span></span></h2>
<p>Gentoostats is written by Vikraman Choudhury as a Google Summer Code 2011 <a rel="nofollow" class="external text" href="https://www.google-melange.com/archive/gsoc/2011/orgs/gentoo/projects/vh4x0r.html">project</a>. It is written in Python and implements a client-server model. The server component is a <a rel="nofollow" class="external text" href="https://www.python.org/dev/peps/pep-0333/">WSGI</a> web application, built using the <a rel="nofollow" class="external text" href="http://webpy.org/">webpy</a> framework. The client component uses the Portage API to collect various statistics from a Gentoo machine, encodes it in the <a rel="nofollow" class="external text" href="https://tools.ietf.org/html/rfc7159">JSON format</a> and submits it to the server. Users have the ability to configure which information is to be transmitted according to their privacy needs.
</p>
<div class="alert alert-danger"><span class="fa fa-fw fa-warning"></span> Warning, this page is a work in progress by <a href="/wiki/User:Gokturk" title="User:Gokturk">gokturk</a> (<a href="/index.php?title=User_talk:Gokturk&action=edit&redlink=1" class="new" title="User talk:Gokturk (page does not exist)">talk</a> | <a href="/wiki/Special:Contributions/gokturk" title="Special:Contributions/gokturk">contribs</a>). Treat its contents with caution.</div>
<h2><span class="mw-headline" id="Suggested_features">Suggested features</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/index.php?title=Project:Gentoostats&veaction=edit&section=2" class="mw-editsection-visualeditor" title="Edit section: Suggested features">edit</a><span class="mw-editsection-divider"> | </span><a href="/index.php?title=Project:Gentoostats&action=edit&section=2" title="Edit section: Suggested features">edit source</a><span class="mw-editsection-bracket">]</span></span></h2>
<h3><span id="Upload_package_build_time_statistics_(veremit)"></span><span class="mw-headline" id="Upload_package_build_time_statistics_.28veremit.29">Upload package build time statistics (veremit)</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/index.php?title=Project:Gentoostats&veaction=edit&section=3" class="mw-editsection-visualeditor" title="Edit section: Upload package build time statistics (veremit)">edit</a><span class="mw-editsection-divider"> | </span><a href="/index.php?title=Project:Gentoostats&action=edit&section=3" title="Edit section: Upload package build time statistics (veremit)">edit source</a><span class="mw-editsection-bracket">]</span></span></h3>
<p>Instead of reporting it in absolute time, look into using a relative measure like SBU (see:<a href="https://en.wikipedia.org/wiki/Linux_From_Scratch#Standard_build_unit" class="extiw" title="wikipedia:Linux From Scratch">SBU</a>).
</p>
<h3><span class="mw-headline" id="Distributed_collection_servers">Distributed collection servers</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/index.php?title=Project:Gentoostats&veaction=edit&section=4" class="mw-editsection-visualeditor" title="Edit section: Distributed collection servers">edit</a><span class="mw-editsection-divider"> | </span><a href="/index.php?title=Project:Gentoostats&action=edit&section=4" title="Edit section: Distributed collection servers">edit source</a><span class="mw-editsection-bracket">]</span></span></h3>
<p>Make multiple servers exchange and sync stats with each other. This is similar to the pgp keyservers and the goal is to distribute the load and combat DDOS. The major problem is the collision of host UUIDs across multiple independent servers. Second problem is the trust between servers for which solutions exist. Since the database will grow significantly large, some form of delta-sync will be necessary.
</p>
<h2><span class="mw-headline" id="Open_problems">Open problems</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/index.php?title=Project:Gentoostats&veaction=edit&section=5" class="mw-editsection-visualeditor" title="Edit section: Open problems">edit</a><span class="mw-editsection-divider"> | </span><a href="/index.php?title=Project:Gentoostats&action=edit&section=5" title="Edit section: Open problems">edit source</a><span class="mw-editsection-bracket">]</span></span></h2>
<h3><span class="mw-headline" id="Validity_of_the_submitted_samples">Validity of the submitted samples</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/index.php?title=Project:Gentoostats&veaction=edit&section=6" class="mw-editsection-visualeditor" title="Edit section: Validity of the submitted samples">edit</a><span class="mw-editsection-divider"> | </span><a href="/index.php?title=Project:Gentoostats&action=edit&section=6" title="Edit section: Validity of the submitted samples">edit source</a><span class="mw-editsection-bracket">]</span></span></h3>
<p>There's no mechanism to stop a malicious user from flooding the server by synthetically creating a large set of statistics reports and submitting them. There's no way to prove that the submitted statistics come from an actual installation. This can be utilized in the form of denial of service or skewing the statistics. Some form of rate limiting and snapshotting may be useful.
</p>
<h2><span class="mw-headline" id="TODO">TODO</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/index.php?title=Project:Gentoostats&veaction=edit&section=7" class="mw-editsection-visualeditor" title="Edit section: TODO">edit</a><span class="mw-editsection-divider"> | </span><a href="/index.php?title=Project:Gentoostats&action=edit&section=7" title="Edit section: TODO">edit source</a><span class="mw-editsection-bracket">]</span></span></h2>
<ul><li>Make an initial release of gentoostats server and add it to the tree</li>
<li>Work with infra about gentoostats deployment</li>
<li>Update <a href="/wiki/Gentoostats" title="Gentoostats">Gentoostats</a>, add a section for deploying private instances, improve the usage text</li></ul>
<h3><span class="mw-headline" id="Discussion_regarding_which_version_to_deploy">Discussion regarding which version to deploy</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/index.php?title=Project:Gentoostats&veaction=edit&section=8" class="mw-editsection-visualeditor" title="Edit section: Discussion regarding which version to deploy">edit</a><span class="mw-editsection-divider"> | </span><a href="/index.php?title=Project:Gentoostats&action=edit&section=8" title="Edit section: Discussion regarding which version to deploy">edit source</a><span class="mw-editsection-bracket">]</span></span></h3>
<table class="table table-striped table-condensed">
<tbody><tr>
<th></th>
<th>Gentoostats 2011</th>
<th>Gentoostats 2012
</th></tr>
<tr>
<td>Pros</td>
<td>
<ul><li>Simplistic design based on web.py with small number of dependencies</li>
<li>With near to no maintenance over 5 years, it was still almost completely functional</li>
<li>Submitting stats is much faster compared to Gentoostats 2012</li></ul>
</td>
<td>
<ul><li>Based on django and has potential for richer web interfaces</li>
<li>Use of django models provides a good data abstraction and makes the solution independent of a particular SQLDB</li></ul>
</td></tr>
<tr>
<td>Cons</td>
<td>
<ul><li>Archaic web interface</li>
<li>Directly deals with SQL queries over web.py, no abstraction</li>
<li>No python3 support for web.py yet</li></ul>
</td>
<td>
<ul><li>Maintenance becomes a burden. Need to keep up with django upgrades.</li>
<li>Submitting statistics works way more slowly</li>
<li>Increased code complexity</li></ul>
</td></tr></tbody></table>
<h3><span class="mw-headline" id="Gentoostats_2011">Gentoostats 2011</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/index.php?title=Project:Gentoostats&veaction=edit&section=9" class="mw-editsection-visualeditor" title="Edit section: Gentoostats 2011">edit</a><span class="mw-editsection-divider"> | </span><a href="/index.php?title=Project:Gentoostats&action=edit&section=9" title="Edit section: Gentoostats 2011">edit source</a><span class="mw-editsection-bracket">]</span></span></h3>
<ul><li>Code: <a rel="nofollow" class="external free" href="https://gitweb.gentoo.org/proj/gentoostats.git/">https://gitweb.gentoo.org/proj/gentoostats.git/</a></li></ul>
<p>Progress reports:
</p>
<ul><li>Progress Report #1: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/2f9044ad5390b53a338fc9bca4bebda5">https://archives.gentoo.org/gentoo-soc/message/2f9044ad5390b53a338fc9bca4bebda5</a></li>
<li>Progress Report #2: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/b345988ca5df929abb4f0f5b9aceb00c">https://archives.gentoo.org/gentoo-soc/message/b345988ca5df929abb4f0f5b9aceb00c</a></li>
<li>Progress Report #3: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/845d373851b7b06f4ab7ce3662b15b4b">https://archives.gentoo.org/gentoo-soc/message/845d373851b7b06f4ab7ce3662b15b4b</a></li>
<li>Progress Report #4: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/76a0eb1b38e9101ca44d5da7723dcf60">https://archives.gentoo.org/gentoo-soc/message/76a0eb1b38e9101ca44d5da7723dcf60</a></li>
<li>Progress Report #5: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/0179caaa96f8df9f4619a38d630c8cdb">https://archives.gentoo.org/gentoo-soc/message/0179caaa96f8df9f4619a38d630c8cdb</a></li>
<li>Midterm Report: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/a982111423d18fb7a714526bf9052708">https://archives.gentoo.org/gentoo-soc/message/a982111423d18fb7a714526bf9052708</a></li>
<li>Progress Report #6: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/635ee0e2c9e3d599be5e9c05cd905f9c">https://archives.gentoo.org/gentoo-soc/message/635ee0e2c9e3d599be5e9c05cd905f9c</a></li>
<li>Progress Report #7: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/606094a198354a2938b8b8a10f7b0cb5">https://archives.gentoo.org/gentoo-soc/message/606094a198354a2938b8b8a10f7b0cb5</a></li>
<li>Final Report: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/c90536fdd571898e6a15c6c7d9fa0c75">https://archives.gentoo.org/gentoo-soc/message/c90536fdd571898e6a15c6c7d9fa0c75</a></li></ul>
<h3><span class="mw-headline" id="Gentoostats_2012">Gentoostats 2012</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/index.php?title=Project:Gentoostats&veaction=edit&section=10" class="mw-editsection-visualeditor" title="Edit section: Gentoostats 2012">edit</a><span class="mw-editsection-divider"> | </span><a href="/index.php?title=Project:Gentoostats&action=edit&section=10" title="Edit section: Gentoostats 2012">edit source</a><span class="mw-editsection-bracket">]</span></span></h3>
<p>Apparently, there's another gentoostats project based on django written as part of GSoC 2012:
</p>
<ul><li>Server: <a rel="nofollow" class="external free" href="https://github.com/gg7/gentoostats_server">https://github.com/gg7/gentoostats_server</a></li>
<li>Client: <a rel="nofollow" class="external free" href="https://github.com/gg7/gentoostats">https://github.com/gg7/gentoostats</a></li>
<li>Playground (??): <a rel="nofollow" class="external free" href="https://github.com/vikraman/gentoostats-playground">https://github.com/vikraman/gentoostats-playground</a></li>
<li>Deployment bug: <a rel="nofollow" class="external free" href="https://bugs.gentoo.org/show_bug.cgi?id=425056">https://bugs.gentoo.org/show_bug.cgi?id=425056</a></li></ul>
<p>Progress reports:
</p>
<ul><li>Progress Report #1: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/a85db0776186d6e4fa032377af2c8634">https://archives.gentoo.org/gentoo-soc/message/a85db0776186d6e4fa032377af2c8634</a></li>
<li>Progress Report #2: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/b0be0d2f6a5c43457ef6cebd3f8e9b7b">https://archives.gentoo.org/gentoo-soc/message/b0be0d2f6a5c43457ef6cebd3f8e9b7b</a></li>
<li>Progress Report #3: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/1b45015692cecc31211f93de4bb701d0">https://archives.gentoo.org/gentoo-soc/message/1b45015692cecc31211f93de4bb701d0</a></li>
<li>Progress Report #4: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/1e1a675494bca49352097a0b25dd58f9">https://archives.gentoo.org/gentoo-soc/message/1e1a675494bca49352097a0b25dd58f9</a></li>
<li>Progress Report #5: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/a8a0f843bd2b755f834b3f9eacdbf97b">https://archives.gentoo.org/gentoo-soc/message/a8a0f843bd2b755f834b3f9eacdbf97b</a></li>
<li>Progress Report #6: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/e8a9ef1386d0bf29d922a86ee5332ea8">https://archives.gentoo.org/gentoo-soc/message/e8a9ef1386d0bf29d922a86ee5332ea8</a></li>
<li>Progress Report #7: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/8e9fcbd3ab67cdc7c66c9aab87eea62f">https://archives.gentoo.org/gentoo-soc/message/8e9fcbd3ab67cdc7c66c9aab87eea62f</a></li>
<li>Final Report: <a rel="nofollow" class="external free" href="https://archives.gentoo.org/gentoo-soc/message/760cbd58a309b56f31d3697d90f44601">https://archives.gentoo.org/gentoo-soc/message/760cbd58a309b56f31d3697d90f44601</a></li></ul>
<p>Find out why the code isn't being hosted on infra. Evaluate the functionality. Determine which version is to be deployed and maintained.
</p>
<h4><span class="mw-headline" id="Attempting_to_deploy_Gentoostats_2012">Attempting to deploy Gentoostats 2012</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/index.php?title=Project:Gentoostats&veaction=edit&section=11" class="mw-editsection-visualeditor" title="Edit section: Attempting to deploy Gentoostats 2012">edit</a><span class="mw-editsection-divider"> | </span><a href="/index.php?title=Project:Gentoostats&action=edit&section=11" title="Edit section: Attempting to deploy Gentoostats 2012">edit source</a><span class="mw-editsection-bracket">]</span></span></h4>
<p>This is an ongoing effort to deploy this version of gentoostats on my local machine:
</p><p>Package list:
</p>
<ul><li>dev-python/django-1.8.9</li>
<li>dev-python/django-extensions-1.6.1</li>
<li>dev-python/django-debug-toolbar-1.3.2</li>
<li>dev-python/django-tastypie-0.9.15</li></ul>
<p>Steps:
</p>
<ul><li>Clone the repo, copy gentoostats/settings.py.example to gentoostats/settings.py and edit accordingly</li>
<li>"manage.py check" dies with "ImportError: No module named south"
<ul><li>south is hard masked</li>
<li>comment out south from INSTALLED_APPS in settings.py</li></ul></li>
<li>"manage.py" check dies with the following:</li></ul>
<pre>File "/usr/lib64/python2.7/site-packages/tastypie/resources.py", line 2256, in ModelResource @transaction.commit_on_success()
AttributeError: 'module' object has no attribute 'commit_on_success'
</pre>
<p>As a hackaroo, edit tastypie and replace "@transaction.commit_on_success" with "@transaction.atomic", see: <a rel="nofollow" class="external free" href="https://github.com/macropin/django-registration/issues/51#issuecomment-100579391">https://github.com/macropin/django-registration/issues/51#issuecomment-100579391</a>
Do the same in gentoostats/receivers/views.py.
</p>
<ul><li>Initialize the database with "manage.py syncdb"</li></ul>
<pre> /usr/lib64/python2.7/site-packages/django/core/management/commands/syncdb.py:24: RemovedInDjango19Warning: The syncdb command will be removed in Django 1.9
</pre>
<ul><li>Run the server with "manage.py runserver"</li></ul>
<pre>January 03, 2017 - 01:01:37
Django version 1.8.9, using settings 'gentoostats.settings'
Starting development server at <a rel="nofollow" class="external free" href="http://127.0.0.1:8000/">http://127.0.0.1:8000/</a>
</pre>
<p>Dies with "ImportError: No module named transaction"
</p>
<ul><li><ul><li>Comment out 'django.middleware.transaction.TransactionMiddleware' from MIDDLEWARE_CLASSES in settings.py, see: <a rel="nofollow" class="external free" href="http://stackoverflow.com/a/33102743">http://stackoverflow.com/a/33102743</a></li></ul></li>
<li>Try to upload stats, dies with:</li></ul>
<pre>INFO 2017-01-03 01:50:31,928 views 27822 140719396648704 process_submission(): Error: Invalid date in LASTSYNC.
Traceback (most recent call last):
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 88, in process_submission
time.strptime(lastsync, "%a, %d %b %Y %H:%M:%S +0000")
File "/usr/lib64/python2.7/_strptime.py", line 478, in _strptime_time
return _strptime(data_string, format)[0]
File "/usr/lib64/python2.7/_strptime.py", line 332, in _strptime
(data_string, format))
ValueError: time data u'Unknown' does not match format '%a, %d %b %Y %H:%M:%S +0000'
</pre>
<p>This is due to using a git repo in /usr/portage, which doesn't contain the timestamp file and the client is sending the string "Unknown" after patching with similarly to <a rel="nofollow" class="external free" href="https://gitweb.gentoo.org/proj/gentoostats.git/commit/?id=963afe1163125b8cbed08c0e8edea9a05a37510e">https://gitweb.gentoo.org/proj/gentoostats.git/commit/?id=963afe1163125b8cbed08c0e8edea9a05a37510e</a>. Patch it with
</p>
<pre>- if lastsync:
+ if lastsync and lastsync != "Unknown":
</pre>
<p>and add the following else statement to it:
</p>
<pre>else:
lastsync = None
</pre>
<ul><li>Try to upload stats again, dies with:</li></ul>
<pre>ERROR 2017-01-03 02:02:24,104 views 810 140115441485568 process_submission(): 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 369, in accept_submission
return process_submission(request)
File "/usr/lib64/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/django/utils/decorators.py", line 145, in inner
return func(*args, **kwargs)
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 159, in process_submission
country = GeoIP().country_name(ip_addr),
File "/usr/lib64/python2.7/site-packages/django/contrib/gis/geoip/base.py", line 190, in country_name
return self.city(query)['country_name']
TypeError: 'NoneType' object has no attribute '__getitem__'
</pre>
<p>Comment out the call to "GeoIP().country_name(ip_addr)" in gentoostats/receiver/view.py for now.
</p>
<ul><li>Try to upload stats again, dies with:</li></ul>
<pre>ERROR 2017-01-03 02:26:40,646 views 13386 140066905986816 process_submission(): Cannot assign "u<i>": "Submission.sync" must be a "SyncServer" instance.</i>
Traceback (most recent call last):
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 369, in accept_submission
return process_submission(request)
File "/usr/lib64/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/django/utils/decorators.py", line 145, in inner
return func(*args, **kwargs)
File "/tmp/gentoostats_server/gentoostats/receiver/views.py", line 186, in process_submission
sync = sync,
File "/usr/lib64/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/django/db/models/query.py", line 346, in create
obj = self.model(**kwargs)
File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", line 468, in __init__
setattr(self, field.name, rel_obj)
File "/usr/lib64/python2.7/site-packages/django/db/models/fields/related.py", line 642, in __set__
self.field.rel.to._meta.object_name,
ValueError: Cannot assign "u<i>": "Submission.sync" must be a "SyncServer" instance.</i>
</pre>
<p>This is due to gentoostats expecting the SYNC variable in make.conf. From gentoostats/stats/models.py:
</p>
<pre># make.conf example: SYNC="rsync://rsync.gentoo.org/gentoo-portage"
sync = models.ForeignKey(SyncServer, blank=True, null=True, related_name='+')
</pre>
<p>Set sync to None for now:
</p>
<pre>@@ -145,6 +147,7 @@ def process_submission(request):
validate_item(lang)
sync = data.get('SYNC')
+ sync = None
if sync:
sync, _ = SyncServer.objects.get_or_create(url=sync)
validate_item(sync)
</pre>
<ul><li>Try to upload stats again, viola!</li></ul>
' |
Unix timestamp of change (timestamp) | 1709431608 |