django-orchestra-test/TODO.md

471 lines
18 KiB
Markdown
Raw Normal View History

2014-11-18 13:59:21 +00:00
==== TODO ====
2014-05-08 16:59:35 +00:00
* use format_html_join for orchestration email alerts
* enforce an emergency email contact and account to contact contacts about problems when mailserver is down
* add `BackendLog` retry action
2015-04-05 18:02:36 +00:00
2014-05-08 16:59:35 +00:00
* webmail identities and addresses
* Permissions .filter_queryset()
* env vars instead of multiple settings files: https://devcenter.heroku.com/articles/config-vars ?
2014-05-27 15:55:09 +00:00
2014-07-24 09:53:34 +00:00
* backend logs with hal logo
2014-08-29 16:13:34 +00:00
2014-09-06 10:56:30 +00:00
* help_text on readonly_fields specialy Bill.state. (eg. A bill is in OPEN state when bla bla )
2014-09-22 15:59:53 +00:00
* order.register_at
@property
def register_on(self):
return order.register_at.date()
2014-09-23 16:23:36 +00:00
2014-09-26 15:05:20 +00:00
* mail backend related_models = ('resources__content_type') ??
2014-09-26 19:21:09 +00:00
* Maildir billing tests/ webdisk billing tests (avg metric)
2014-09-28 12:28:57 +00:00
2015-03-11 20:01:08 +00:00
* when using modeladmin to store shit like self.account, make sure to have a cleanslate in each request? no, better reuse the last one
2014-09-28 12:28:57 +00:00
2015-03-11 20:01:08 +00:00
* jabber with mailbox accounts (dovecot mail notification)
2014-09-28 12:28:57 +00:00
2014-11-16 18:39:31 +00:00
* rename accounts register to "account", and reated api and admin references
2014-09-28 12:28:57 +00:00
2015-03-11 20:01:08 +00:00
* AccountAdminMixin auto adds 'account__name' on searchfields
* What fields we really need on contacts? name email phone and what more?
2014-09-30 14:46:29 +00:00
2014-10-03 14:02:11 +00:00
* DOC: Complitely decouples scripts execution, billing, service definition
* init.d celery scripts
-# Required-Start: $network $local_fs $remote_fs postgresql celeryd
-# Required-Stop: $network $local_fs $remote_fs postgresql celeryd
2014-10-15 21:18:50 +00:00
* regenerate virtual_domains every time (configure a separate file for orchestra on postfix)
2014-10-17 10:04:47 +00:00
* Backend optimization
* fields = ()
* ignore_fields = ()
* based on a merge set of save(update_fields)
2014-10-17 20:03:41 +00:00
2014-10-23 15:38:46 +00:00
* proforma without billing contact?
2015-03-11 20:01:08 +00:00
* print open invoices as proforma?
* env ORCHESTRA_MASTER_SERVER='test1.orchestra.lan' ORCHESTRA_SECOND_SERVER='test2.orchestra.lan' ORCHESTRA_SLAVE_SERVER='test3.orchestra.lan' python3 manage.py test orchestra.contrib.domains.tests.functional_tests.tests:AdminBind9BackendDomainTest --nologcapture --keepdb
2014-10-23 15:38:46 +00:00
* ForeignKey.swappable
* REST PERMISSIONS
2014-10-24 10:16:46 +00:00
2014-11-02 14:33:55 +00:00
* Databases.User add reverse M2M databases widget (like mailbox.addresses)
2014-11-05 20:22:01 +00:00
* Make one dedicated CGI user for each account only for CGI execution (fpm/fcgid). Different from the files owner, and without W permissions, so attackers can not inject backdors and malware.
* resource min max allocation with validation
2014-11-09 10:16:07 +00:00
* domain validation parse named-checzone output to assign errors to fields
* Directory Protection on webapp and use webapp path as base path (validate)
* webapp backend option compatibility check? raise exception, missconfigured error
2014-11-18 13:59:21 +00:00
* Resource used_list_display=True, allocated_list_displat=True, allow resources to show up on list_display
* BackendLog.updated_at (tasks that run over several minutes when finished they do not appear first on the changelist) (like celery tasks.when)
2014-11-20 15:34:59 +00:00
* Create an admin service_view with icons (like SaaS app)
2015-02-24 09:34:26 +00:00
* prevent @pangea.org email addresses on contacts, enforce at least one email without @pangea.org
2014-12-22 11:40:02 +00:00
2015-03-01 11:56:54 +00:00
ln -s /proc/self/fd /dev/fd
POST INSTALL
------------
* Generate a password-less ssh key, and copy it to the servers you want to orchestrate.
ssh-keygen
ssh-copy-id root@<server-address>
Php binaries should have this format: /usr/bin/php5.2-cgi
2015-03-04 21:06:16 +00:00
* logs on panel/logs/ ? mkdir ~webapps, backend post save signal?
* <IfModule security2_module> and other IfModule on backend SecRule
2015-10-07 22:05:00 +00:00
# Orchestra global search box on the page head, based https://github.com/django/django/blob/master/django/contrib/admin/options.py#L866 and iterating over all registered services and inspectin its admin.search_fields
2015-03-11 20:01:08 +00:00
2015-03-25 17:04:44 +00:00
* contain error on plugin missing key (plugin dissabled): NOP, fail hard is better than silently, perhaps fail at starttime? apploading machinary
2015-03-12 14:05:23 +00:00
* contact.alternative_phone on a phone.tooltip, email:to
* make sure that you understand the risks
2015-03-18 21:51:12 +00:00
* full support for deactivation of services/accounts
2015-03-25 17:04:44 +00:00
* Display admin.is_active (disabled account special icon and order by support)
2015-03-18 21:51:12 +00:00
* lock resource monitoring
* -EXecCGI in common CMS upload locations /wp-upload/upload/uploads
* cgi user / pervent shell access
* prevent stderr when users exists on backend i.e. mysql user create
* disable anonymized list options (mailman)
2015-03-23 15:36:51 +00:00
* tags = GenericRelation(TaggedItem, related_query_name='bookmarks')
* user provided crons
* ```<?php
$moodle_host = $SERVER[HTTP_HOST];
require_once(/etc/moodles/.$moodle_host.config.php);``` moodle/drupla/php-list multi-tenancy
* make account available on all admin forms
2015-03-27 19:50:54 +00:00
* more robust backend error handling, continue executing but exit code > 0 if failure: failing_cmd || exit_code=1 and don't forget to call super.commit()!!
2015-03-23 15:36:51 +00:00
2015-03-27 19:50:54 +00:00
* website directives uniquenes validation on serializers
2015-03-27 19:50:54 +00:00
+ is_Active custom filter with support for instance.account.is_Active annotate with F() needed (django 1.8)
2015-03-27 19:50:54 +00:00
* document service help things: discount/refound/compensation effect and metric table
* Document metric interpretation help_text
* document plugin serialization, data_serializer?
2015-10-07 22:05:00 +00:00
* Document strong input validation
2015-03-27 19:50:54 +00:00
2015-04-05 18:02:36 +00:00
# bill line managemente, remove, undo (only when possible), move, copy, paste
2015-03-27 19:50:54 +00:00
* budgets: no undo feature
* Autocomplete admin fields like <site_name>.phplist... with js
* allow empty metric pack for default rates? changes on rating algo
2015-03-29 16:10:07 +00:00
* payment methods icons
* use server.name | server.address on python backends, like gitlab instead of settings?
* TODO raise404, here and everywhere
* update service orders on a celery task? because it take alot
2015-03-29 16:10:07 +00:00
2015-04-03 10:14:45 +00:00
# FIXME do more test, make sure billed until doesn't get uodated whhen services are billed with les metric, and don't upgrade billed_until when undoing under this circumstances
# * line 513: change threshold and one time service metric change should update last value if not billed, only record for recurring invoicing. postpay services should store the last metric for pricing period.
# * add ini, end dates on bill lines and breakup quanity into size(defaut:1) and metric
# * threshold for significative metric accountancy on services.handler
# * http://orchestra.pangea.org/admin/orders/order/6418/
2015-03-29 16:10:07 +00:00
* move normurlpath to orchestra.utils from websites.utils
* write down insights
* websites directives get_location() and use it on last change view validation stage to compare with contents.location and also on the backend ?
* modeladmin Default filter + search isn't working, prepend filter when searching
2015-04-05 18:02:36 +00:00
* create service help templates based on urlqwargs with the most basic services.
2015-03-29 16:10:07 +00:00
Translation
-----------
mkdir locale
django-admin.py makemessages -l ca
django-admin.py compilemessages -l ca
https://docs.djangoproject.com/en/1.7/topics/i18n/translation/#joining-strings-string-concat
from django.utils.translation import ugettext
from django.utils import translation
translation.activate('ca')
ugettext("Description")
* saas validate_creation generic approach, for all backends. standard output
2015-03-29 16:10:07 +00:00
# create orchestrate databases.Database pk=1 -n --dry-run | --noinput --action save (default)|delete --backend name (limit to this backend) --help
2015-04-01 15:49:21 +00:00
* postupgradeorchestra send signals in order to hook custom stuff
2015-04-03 10:14:45 +00:00
* gevent is not ported to python3 :'(
2015-04-04 17:44:07 +00:00
# FIXME account deletion generates an integrity error
https://code.djangoproject.com/ticket/24576
2015-04-03 10:14:45 +00:00
# FIXME what to do when deleting accounts? set fk null and fill a username charfield? issues, invoices.. we whant all this to go away?
* implement delete All related services
2015-04-03 13:03:08 +00:00
2015-04-04 17:44:07 +00:00
* read https://docs.djangoproject.com/en/dev/releases/1.8/ and fix deprecation warnings
2015-04-29 10:51:30 +00:00
* create nice fieldsets for SaaS, WebApp types and services, and helptexts too!
2015-04-04 17:44:07 +00:00
* replace make_option in management commands
2015-04-05 18:02:36 +00:00
# FIXME model contact info and account info (email, name, etc) correctly/unredundant/dry
* Use the new django.contrib.admin.RelatedOnlyFieldListFilter in ModelAdmin.list_filter to limit the list_filter choices to foreign objects which are attached to those from the ModelAdmin.
+ Query Expressions, Conditional Expressions, and Database Functions¶
* forms: You can now pass a callable that returns an iterable of choices when instantiating a ChoiceField.
2015-04-07 15:14:49 +00:00
* move all tests to django-orchestra/tests
* *natural keys: those fields that uniquely identify a service, list.name, website.name, webapp.name+account, make sure rest api can not edit thos things
2015-04-08 14:41:09 +00:00
2015-04-09 14:32:10 +00:00
* MultiCHoiceField proper serialization
* replace unique_name by natural_key?
* do not require contact or create default
2015-04-20 14:23:10 +00:00
* abstract model classes that enabling overriding, and ORCHESTRA_DATABASE_MODEL settings + orchestra.get_database_model() instead of explicitly importing from orchestra.contrib.databases.models import Database.. (Admin and REST API are fucked then?)
2015-04-14 14:29:22 +00:00
2015-04-24 11:39:20 +00:00
# billing order list filter detect metrics that are greater from those of billing_date
2015-04-16 13:15:21 +00:00
# Ignore superusers & co on billing: list filter doesn't work nor ignore detection
2015-04-14 14:29:22 +00:00
# bill.totals make it 100% computed?
2015-04-16 13:15:21 +00:00
* joomla: wget https://github.com/joomla/joomla-cms/releases/download/3.4.1/Joomla_3.4.1-Stable-Full_Package.tar.gz -O - | tar xvfz -
2015-04-20 14:23:10 +00:00
2015-04-21 13:12:48 +00:00
# Amend lines???
2015-04-27 14:54:17 +00:00
# orders currency setting
2015-04-20 14:23:10 +00:00
2015-04-23 19:46:23 +00:00
# Determine the difference between data serializer used for validation and used for the rest API!
# Make PluginApiView that fills metadata and other stuff like modeladmin plugin support
2015-04-26 13:53:00 +00:00
2015-05-07 19:00:02 +00:00
# reset setting button
2015-04-29 10:51:30 +00:00
# admin edit relevant djanog settings
# django SITE_NAME vs ORCHESTRA_SITE_NAME ?
2015-04-29 21:35:56 +00:00
2015-04-30 11:24:18 +00:00
2015-05-04 19:52:53 +00:00
# TASKS_ENABLE_UWSGI_CRON_BEAT (default) for production + system check --deploy
if 'wsgi' in sys.argv and settings.TASKS_ENABLE_UWSGI_CRON_BEAT:
import uwsgi
def uwsgi_beat(signum):
print "It's 5 o'clock of the first day of the month."
uwsgi.register_signal(99, '', uwsgi_beat)
uwsgi.add_timer(99, 60)
# TASK_BEAT_BACKEND = ('cron', 'celerybeat', 'uwsgi')
2015-05-07 19:00:02 +00:00
# Ship orchestra production-ready (no DEBUG etc)
2015-05-04 19:52:53 +00:00
2015-05-03 17:44:46 +00:00
# reload generic admin view ?redirect=http...
2015-05-04 19:52:53 +00:00
# inspecting django db connection for asserting db readines? or performing a query
2015-08-05 22:58:35 +00:00
* wake up django mailer on send_mail
2015-05-03 17:44:46 +00:00
2015-05-03 21:26:17 +00:00
from orchestra.contrib.tasks import task
import time, sys
@task(name='rata')
def counter(num, log):
for i in range(1, num):
with open(log, 'a') as handler:
handler.write(str(i))
sys.stderr.write('hola\n')
time.sleep(1)
counter.apply_async(10, '/tmp/kakas')
2015-08-05 22:58:35 +00:00
* Provide some fixtures with mocked data
2015-05-04 10:48:09 +00:00
2015-05-04 12:57:41 +00:00
TODO http://wiki2.dovecot.org/HowTo/SimpleVirtualInstall
TODO http://wiki2.dovecot.org/HowTo/VirtualUserFlatFilesPostfix
TODO mount the filesystem with "nosuid" option
2015-05-07 19:00:02 +00:00
2015-08-05 22:58:35 +00:00
* uwse uwsgi cron: decorator or config cron = 59 2 -1 -1 -1 %(virtualenv)/bin/python manage.py runmyfunnytask
2015-05-04 22:07:14 +00:00
2015-05-05 19:42:55 +00:00
# mailboxes.address settings multiple local domains, not only one?
2015-05-07 19:00:02 +00:00
# backend.context = self.get_context() or save(obj, context=None) ?? more like form.cleaned_data
2015-05-05 19:42:55 +00:00
# smtplib.SMTPConnectError: (421, b'4.7.0 mail.pangea.org Error: too many connections from 77.246.181.209')
# rename virtual_maps to virtual_alias_maps and remove virtual_alias_domains ?
2015-08-05 22:58:35 +00:00
# virtdomains file is not ideal, prevent user provided fake/error domains there! and make sure to chekc if this file is required!
2015-05-06 09:48:09 +00:00
2015-05-05 20:55:54 +00:00
# Deprecate restart/start/stop services (do touch wsgi.py and fuck celery)
orchestra-beat support for uwsgi cron
make django admin taskstate uncollapse fucking traceback, ( if exists ?)
2015-05-07 19:00:02 +00:00
# form for custom message on admin save "comment & save"?
# backend.context and backned.instance provided when an action is called? like forms.cleaned_data: do it on manager.generation(backend.context = backend.get_context()) or in backend.__getattr__ ? also backend.head,tail,content switching on manager.generate()?
2015-05-09 17:08:45 +00:00
resorce monitoring more efficient, less mem an better queries for calc current data
2015-05-09 18:53:23 +00:00
2015-05-12 14:04:20 +00:00
# bill this https://orchestra.pangea.org/admin/orders/order/8236/ should be already billed, <= vs <
# Convert rating method from function to PluginClass
2015-05-13 12:16:51 +00:00
# autoresponses on mailboxes, not addresses or remove them
2015-05-13 13:52:20 +00:00
# force save and continue on routes (and others?)
2015-05-13 17:22:54 +00:00
# gevent for python3
2015-05-13 14:27:24 +00:00
apt-get install cython3
export CYTHON='cython3'
pip3 install https://github.com/fantix/gevent/archive/master.zip
# SIgnal handler for notify workers to reload stuff, like resource sync: https://docs.python.org/2/library/signal.html
2015-05-15 14:19:24 +00:00
# BUG Delete related services also deletes account!
2015-05-18 15:21:42 +00:00
# get_related service__rates__isnull=TRue is that correct?
# uwsgi hot reload? http://uwsgi-docs.readthedocs.org/en/latest/articles/TheArtOfGracefulReloading.html
# change mailer.message.priority by, queue/sent inmediatelly or rename critical to noq
2015-05-19 13:27:04 +00:00
method(
2015-05-18 15:21:42 +00:00
arg, arg, arg)
2016-03-08 10:16:49 +00:00
Bash/Python/PHPController
2015-05-22 13:15:06 +00:00
# services.handler as generator in order to save memory? not swell like a balloon
import uwsgi
from uwsgidecorators import timer
from django.utils import autoreload
@timer(3)
def change_code_gracefull_reload(sig):
if autoreload.code_changed():
uwsgi.reload()
# using kill to send the signal
kill -HUP `cat /tmp/project-master.pid`
# or the convenience option --reload
uwsgi --reload /tmp/project-master.pid
# or if uwsgi was started with touch-reload=/tmp/somefile
touch /tmp/somefile
2015-05-30 14:44:05 +00:00
# Serializers.validation migration to DRF3: grep -r 'attrs, source' *|grep -v '~'
serailzer self.instance on create.
2015-06-22 14:14:16 +00:00
* check certificate: websites directive ssl + domains search on miscellaneous
2015-06-09 11:16:36 +00:00
2015-06-22 14:14:16 +00:00
# billing invoice link on related invoices not overflow nginx GET vars
* backendLog store method and language... and use it for display_script with correct lexer
2015-07-08 13:29:29 +00:00
@register.filter
def comma(value):
value = str(value)
if '.' in value:
left, right = str(value).split('.')
return ','.join((left, right))
return value
2015-07-10 13:00:51 +00:00
# payment/bill report allow to change template using a setting variable
2015-07-21 12:23:40 +00:00
# Payment transaction stats, graphs over time
2015-07-13 11:31:32 +00:00
reporter.stories_filed = F('stories_filed') + 1
reporter.save()
In order to access the new value that has been saved in this way, the object will need to be reloaded:
https://docs.djangoproject.com/en/dev/ref/models/conditional-expressions/
Greatest
Colaesce('total', 'computed_total')
Case
# SQL case on payment transaction state ? case when trans.amount >
2015-07-28 10:49:20 +00:00
# Resource inline links point to custom changelist view that preserve state (breadcrumbs, title, etc) rather than generic changeview with queryarg filtering
# ORDER diff Pending vs ALL
# DELETING RESOURCE RELATED OBJECT SHOULD NOT delete related monitor data for traffic accountancy
2015-07-29 09:05:07 +00:00
# round decimals on every billing operation
2015-07-29 09:05:07 +00:00
# use "su $user --shell /bin/bash" on backends for security : MKDIR -p...
# model.field.flatchoices
2015-08-05 22:58:35 +00:00
* This is beta software, please test thoroughly before putting into production and report back any issues.
# messages SMTP errors: temporary->deferre else Failed
# Don't enforce one contact per account? remove account.email in favour of contacts?
2015-08-31 11:58:59 +00:00
# Mailer: mark as sent
# Mailer: download attachments
2015-09-21 10:28:49 +00:00
# Enable/disable ignore period orders list filter
2015-09-23 12:22:32 +00:00
# Modsecurity rules template by cms (wordpress, joomla, dokuwiki (973337 973338 973347 958057), ...
2015-09-30 20:33:25 +00:00
deploy --dev
deploy.sh and deploy-dev.sh autoupgrade
2015-09-30 20:33:25 +00:00
short URLS: https://github.com/rsvp/gitio
link backend help text variables to settings/#var_name
2015-10-01 18:02:23 +00:00
mkhomedir_helper or create ssh homes with bash.rc and such
2015-10-02 09:31:32 +00:00
# warnings if some plugins are disabled, like make routes red
# replace show emails by https://docs.python.org/3/library/email.contentmanager.html#module-email.contentmanager
2015-10-05 12:09:11 +00:00
2015-10-07 13:15:16 +00:00
# setupforbiddendomains --url alexa -n 5000
2015-10-07 22:05:00 +00:00
* remove welcome box on dashboard?
# account contacts inline, show provided fields and ignore the rest?
# email usage -webkit-column-count:3;-moz-column-count:3;column-count:3;
2015-10-08 13:54:39 +00:00
2015-10-09 12:54:30 +00:00
# validate_user on saas.wordpress to detect if username already exists before attempting to create a blog
2015-10-15 22:31:54 +00:00
# webapps don't override owner and permissions on every save(), just on create
# webapps php fpm allow pool config to be overriden. template + pool inheriting template?
# get_context signal to overridaconfiguration? best practice: all context on get_context, ever use other context. template rendering as backend generator: proof of concept
2016-02-23 11:49:10 +00:00
# if not database_ready(): schedule a retry in 60 seconds, otherwise resources and other dynamic content gets fucked, maybe attach some 'signal' when first query goes trough
2015-10-15 22:31:54 +00:00
with database_ready:
shit_happend, otherwise schedule for first query
# Entry.objects.filter()[:1].first() (LIMIT 1)
2015-10-29 18:19:00 +00:00
2015-11-03 11:09:04 +00:00
2015-12-02 18:53:20 +00:00
# Reverse lOgHistory order by date (lastest first)
* setuppostgres use porject_name for db name and user instead of orchestra
2016-02-09 12:17:42 +00:00
# POSTFIX web traffic monitor '": uid=" from=<%(user)s>'
2016-02-09 12:17:42 +00:00
2016-02-11 14:24:09 +00:00
# Automatically re-run backends until success? only timedout executions?
2016-02-16 09:47:08 +00:00
# TODO save serialized versions ob backendoperation.instance in order to allow backend reexecution of deleted objects
2016-02-09 12:17:42 +00:00
2016-03-11 12:19:34 +00:00
# lets encrypt: DNS vs HTTP challange
2016-03-11 14:20:28 +00:00
# lets enctypt: autorenew
2016-03-11 12:19:34 +00:00
# Warning websites with ssl options without https protocol
# Schedule cancellation
2016-03-31 16:02:50 +00:00
# Multiple domains wordpress
# Reversion
2016-04-06 19:00:16 +00:00
# Disable/enable SaaS and VPS
# Don't show lines with size 0?
# pending orders with recharge do not show up
# Traffic of disabled accounts doesn't get disabled
# URL encode "Order description" on clone
# Service CLONE METRIC doesn't work
# Show warning when saving order and metricstorage date is inconistent with registered date!
# exclude from change list action, support for multiple exclusion
2016-04-30 12:01:29 +00:00
# breadcrumbs https://orchestra.pangea.org/admin/domains/domain/?account_id=930
2016-05-07 10:32:51 +00:00
with open(file) as handler:
os.unlink(file)
# Mark transaction process as executed should not override higher transaction states
2016-05-18 14:08:12 +00:00
# Bill amend and related transaction, what to do? allow edit transaction ammount of amends when their are pending execution
2016-05-18 14:08:12 +00:00
# DASHBOARD: Show owned tickets, scheduled actions, maintenance operations (diff domains)
2016-06-17 10:00:04 +00:00
# Add confirmation step on transaction actions like process transaction
# SAVE INISTIAL PASSWORD from all services, and just use it to create the service, never update it
# Don't use system groups for unixmailbackends