django-orchestra-test/TODO.md

587 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

==== TODO ====
* use format_html_join for orchestration email alerts
* enforce an emergency email contact and account to contact contacts about problems when mailserver is down
* add `BackendLog` retry action
* webmail identities and addresses
* Permissions .filter_queryset()
* env vars instead of multiple settings files: https://devcenter.heroku.com/articles/config-vars ?
* backend logs with hal logo
* help_text on readonly_fields specialy Bill.state. (eg. A bill is in OPEN state when bla bla )
* order.register_at
@property
def register_on(self):
return order.register_at.date()
* mail backend related_models = ('resources__content_type') ??
* Maildir billing tests/ webdisk billing tests (avg metric)
* when using modeladmin to store shit like self.account, make sure to have a cleanslate in each request? no, better reuse the last one
* jabber with mailbox accounts (dovecot mail notification)
* rename accounts register to "account", and reated api and admin references
* AccountAdminMixin auto adds 'account__name' on searchfields
* What fields we really need on contacts? name email phone and what more?
* DOC: Complitely decouples scripts execution, billing, service definition
* init.d celery scripts
-# Required-Start: $network $local_fs $remote_fs postgresql celeryd
-# Required-Stop: $network $local_fs $remote_fs postgresql celeryd
* regenerate virtual_domains every time (configure a separate file for orchestra on postfix)
* Backend optimization
* fields = ()
* ignore_fields = ()
* based on a merge set of save(update_fields)
* proforma without billing contact?
* print open invoices as proforma?
* env ORCHESTRA_MASTER_SERVER='test1.orchestra.lan' ORCHESTRA_SECOND_SERVER='test2.orchestra.lan' ORCHESTRA_SLAVE_SERVER='test3.orchestra.lan' python3 manage.py test orchestra.contrib.domains.tests.functional_tests.tests:AdminBind9BackendDomainTest --nologcapture --keepdb
* ForeignKey.swappable
* REST PERMISSIONS
* Databases.User add reverse M2M databases widget (like mailbox.addresses)
* Make one dedicated CGI user for each account only for CGI execution (fpm/fcgid). Different from the files owner, and without W permissions, so attackers can not inject backdors and malware.
* resource min max allocation with validation
* domain validation parse named-checzone output to assign errors to fields
* Directory Protection on webapp and use webapp path as base path (validate)
* webapp backend option compatibility check? raise exception, missconfigured error
* Resource used_list_display=True, allocated_list_displat=True, allow resources to show up on list_display
* BackendLog.updated_at (tasks that run over several minutes when finished they do not appear first on the changelist) (like celery tasks.when)
* Create an admin service_view with icons (like SaaS app)
* prevent @pangea.org email addresses on contacts, enforce at least one email without @pangea.org
ln -s /proc/self/fd /dev/fd
POST INSTALL
------------
* Generate a password-less ssh key, and copy it to the servers you want to orchestrate.
ssh-keygen
ssh-copy-id root@<server-address>
Php binaries should have this format: /usr/bin/php5.2-cgi
* logs on panel/logs/ ? mkdir ~webapps, backend post save signal?
* <IfModule security2_module> and other IfModule on backend SecRule
# Orchestra global search box on the page head, based https://github.com/django/django/blob/master/django/contrib/admin/options.py#L866 and iterating over all registered services and inspectin its admin.search_fields
* contain error on plugin missing key (plugin dissabled): NOP, fail hard is better than silently, perhaps fail at starttime? apploading machinary
* contact.alternative_phone on a phone.tooltip, email:to
* make sure that you understand the risks
* full support for deactivation of services/accounts
* Display admin.is_active (disabled account special icon and order by support)
* lock resource monitoring
* -EXecCGI in common CMS upload locations /wp-upload/upload/uploads
* cgi user / pervent shell access
* prevent stderr when users exists on backend i.e. mysql user create
* disable anonymized list options (mailman)
* tags = GenericRelation(TaggedItem, related_query_name='bookmarks')
* user provided crons
* ```<?php
$moodle_host = $SERVER[HTTP_HOST];
require_once(/etc/moodles/.$moodle_host.config.php);``` moodle/drupla/php-list multi-tenancy
* make account available on all admin forms
* more robust backend error handling, continue executing but exit code > 0 if failure: failing_cmd || exit_code=1 and don't forget to call super.commit()!!
* website directives uniquenes validation on serializers
+ is_Active custom filter with support for instance.account.is_Active annotate with F() needed (django 1.8)
* document service help things: discount/refound/compensation effect and metric table
* Document metric interpretation help_text
* document plugin serialization, data_serializer?
* Document strong input validation
# bill line managemente, remove, undo (only when possible), move, copy, paste
* budgets: no undo feature
* Autocomplete admin fields like <site_name>.phplist... with js
* allow empty metric pack for default rates? changes on rating algo
* payment methods icons
* use server.name | server.address on python backends, like gitlab instead of settings?
* TODO raise404, here and everywhere
* update service orders on a celery task? because it take alot
# FIXME do more test, make sure billed until doesn't get uodated whhen services are billed with les metric, and don't upgrade billed_until when undoing under this circumstances
# * line 513: change threshold and one time service metric change should update last value if not billed, only record for recurring invoicing. postpay services should store the last metric for pricing period.
# * add ini, end dates on bill lines and breakup quanity into size(defaut:1) and metric
# * threshold for significative metric accountancy on services.handler
# * http://orchestra.pangea.org/admin/orders/order/6418/
* move normurlpath to orchestra.utils from websites.utils
* write down insights
* websites directives get_location() and use it on last change view validation stage to compare with contents.location and also on the backend ?
* modeladmin Default filter + search isn't working, prepend filter when searching
* create service help templates based on urlqwargs with the most basic services.
Translation
-----------
mkdir locale
django-admin.py makemessages -l ca
django-admin.py compilemessages -l ca
https://docs.djangoproject.com/en/1.7/topics/i18n/translation/#joining-strings-string-concat
from django.utils.translation import ugettext
from django.utils import translation
translation.activate('ca')
ugettext("Description")
* saas validate_creation generic approach, for all backends. standard output
# create orchestrate databases.Database pk=1 -n --dry-run | --noinput --action save (default)|delete --backend name (limit to this backend) --help
* postupgradeorchestra send signals in order to hook custom stuff
* gevent is not ported to python3 :'(
# FIXME account deletion generates an integrity error
https://code.djangoproject.com/ticket/24576
# FIXME what to do when deleting accounts? set fk null and fill a username charfield? issues, invoices.. we whant all this to go away?
* implement delete All related services
* read https://docs.djangoproject.com/en/dev/releases/1.8/ and fix deprecation warnings
* create nice fieldsets for SaaS, WebApp types and services, and helptexts too!
* replace make_option in management commands
# FIXME model contact info and account info (email, name, etc) correctly/unredundant/dry
* Use the new django.contrib.admin.RelatedOnlyFieldListFilter in ModelAdmin.list_filter to limit the list_filter choices to foreign objects which are attached to those from the ModelAdmin.
+ Query Expressions, Conditional Expressions, and Database Functions¶
* forms: You can now pass a callable that returns an iterable of choices when instantiating a ChoiceField.
* move all tests to django-orchestra/tests
* *natural keys: those fields that uniquely identify a service, list.name, website.name, webapp.name+account, make sure rest api can not edit thos things
* MultiCHoiceField proper serialization
* replace unique_name by natural_key?
* do not require contact or create default
* abstract model classes that enabling overriding, and ORCHESTRA_DATABASE_MODEL settings + orchestra.get_database_model() instead of explicitly importing from orchestra.contrib.databases.models import Database.. (Admin and REST API are fucked then?)
# billing order list filter detect metrics that are greater from those of billing_date
# Ignore superusers & co on billing: list filter doesn't work nor ignore detection
# bill.totals make it 100% computed?
* joomla: wget https://github.com/joomla/joomla-cms/releases/download/3.4.1/Joomla_3.4.1-Stable-Full_Package.tar.gz -O - | tar xvfz -
# Amend lines???
# orders currency setting
# Determine the difference between data serializer used for validation and used for the rest API!
# Make PluginApiView that fills metadata and other stuff like modeladmin plugin support
# reset setting button
# admin edit relevant djanog settings
# django SITE_NAME vs ORCHESTRA_SITE_NAME ?
# TASKS_ENABLE_UWSGI_CRON_BEAT (default) for production + system check --deploy
if 'wsgi' in sys.argv and settings.TASKS_ENABLE_UWSGI_CRON_BEAT:
import uwsgi
def uwsgi_beat(signum):
print "It's 5 o'clock of the first day of the month."
uwsgi.register_signal(99, '', uwsgi_beat)
uwsgi.add_timer(99, 60)
# TASK_BEAT_BACKEND = ('cron', 'celerybeat', 'uwsgi')
# Ship orchestra production-ready (no DEBUG etc)
# reload generic admin view ?redirect=http...
# inspecting django db connection for asserting db readines? or performing a query
* wake up django mailer on send_mail
from orchestra.contrib.tasks import task
import time, sys
@task(name='rata')
def counter(num, log):
for i in range(1, num):
with open(log, 'a') as handler:
handler.write(str(i))
sys.stderr.write('hola\n')
time.sleep(1)
counter.apply_async(10, '/tmp/kakas')
* Provide some fixtures with mocked data
TODO http://wiki2.dovecot.org/HowTo/SimpleVirtualInstall
TODO http://wiki2.dovecot.org/HowTo/VirtualUserFlatFilesPostfix
TODO mount the filesystem with "nosuid" option
* uwse uwsgi cron: decorator or config cron = 59 2 -1 -1 -1 %(virtualenv)/bin/python manage.py runmyfunnytask
# mailboxes.address settings multiple local domains, not only one?
# backend.context = self.get_context() or save(obj, context=None) ?? more like form.cleaned_data
# smtplib.SMTPConnectError: (421, b'4.7.0 mail.pangea.org Error: too many connections from 77.246.181.209')
# rename virtual_maps to virtual_alias_maps and remove virtual_alias_domains ?
# virtdomains file is not ideal, prevent user provided fake/error domains there! and make sure to chekc if this file is required!
# Deprecate restart/start/stop services (do touch wsgi.py and fuck celery)
orchestra-beat support for uwsgi cron
make django admin taskstate uncollapse fucking traceback, ( if exists ?)
# form for custom message on admin save "comment & save"?
# backend.context and backned.instance provided when an action is called? like forms.cleaned_data: do it on manager.generation(backend.context = backend.get_context()) or in backend.__getattr__ ? also backend.head,tail,content switching on manager.generate()?
resorce monitoring more efficient, less mem an better queries for calc current data
# bill this https://orchestra.pangea.org/admin/orders/order/8236/ should be already billed, <= vs <
# Convert rating method from function to PluginClass
# autoresponses on mailboxes, not addresses or remove them
# force save and continue on routes (and others?)
# gevent for python3
apt-get install cython3
export CYTHON='cython3'
pip3 install https://github.com/fantix/gevent/archive/master.zip
# SIgnal handler for notify workers to reload stuff, like resource sync: https://docs.python.org/2/library/signal.html
# BUG Delete related services also deletes account!
# get_related service__rates__isnull=TRue is that correct?
# uwsgi hot reload? http://uwsgi-docs.readthedocs.org/en/latest/articles/TheArtOfGracefulReloading.html
# change mailer.message.priority by, queue/sent inmediatelly or rename critical to noq
method(
arg, arg, arg)
Bash/Python/PHPBackend
# services.handler as generator in order to save memory? not swell like a balloon
import uwsgi
from uwsgidecorators import timer
from django.utils import autoreload
@timer(3)
def change_code_gracefull_reload(sig):
if autoreload.code_changed():
uwsgi.reload()
# using kill to send the signal
kill -HUP `cat /tmp/project-master.pid`
# or the convenience option --reload
uwsgi --reload /tmp/project-master.pid
# or if uwsgi was started with touch-reload=/tmp/somefile
touch /tmp/somefile
# Serializers.validation migration to DRF3: grep -r 'attrs, source' *|grep -v '~'
serailzer self.instance on create.
* check certificate: websites directive ssl + domains search on miscellaneous
# billing invoice link on related invoices not overflow nginx GET vars
* backendLog store method and language... and use it for display_script with correct lexer
@register.filter
def comma(value):
value = str(value)
if '.' in value:
left, right = str(value).split('.')
return ','.join((left, right))
return value
# payment/bill report allow to change template using a setting variable
# Payment transaction stats, graphs over time
reporter.stories_filed = F('stories_filed') + 1
reporter.save()
In order to access the new value that has been saved in this way, the object will need to be reloaded:
https://docs.djangoproject.com/en/dev/ref/models/conditional-expressions/
Greatest
Colaesce('total', 'computed_total')
Case
# SQL case on payment transaction state ? case when trans.amount >
# Resource inline links point to custom changelist view that preserve state (breadcrumbs, title, etc) rather than generic changeview with queryarg filtering
# ORDER diff Pending vs ALL
# DELETING RESOURCE RELATED OBJECT SHOULD NOT delete related monitor data for traffic accountancy
# round decimals on every billing operation
# use "su $user --shell /bin/bash" on backends for security : MKDIR -p...
# model.field.flatchoices
* This is beta software, please test thoroughly before putting into production and report back any issues.
# messages SMTP errors: temporary->deferre else Failed
# Don't enforce one contact per account? remove account.email in favour of contacts?
# Mailer: mark as sent
# Mailer: download attachments
# Enable/disable ignore period orders list filter
# Modsecurity rules template by cms (wordpress, joomla, dokuwiki (973337 973338 973347 958057), ...
deploy --dev
deploy.sh and deploy-dev.sh autoupgrade
short URLS: https://github.com/rsvp/gitio
link backend help text variables to settings/#var_name
make python3 default python on the fucking docker container
autocomplete; on the form header and type="search"
To latest developers to post on this thread: I implemented the workaround I described in comment #14 nearly three months ago, and it has worked perfectly since then. While we would all prefer that "autocomplete=off" function properly at all times, it still functions properly if you include in your form an input element with any other autocomplete value.
I simply added this code to my layout:
<div style="display: none;">
<input type="text" id="PreventChromeAutocomplete" name="PreventChromeAutocomplete" autocomplete="address-level4" />
</div>
Once I did this, all of my "autocomplete=off" elements were respected by Chrome.
<input type="password" name="password" value="" style="display: none" />
http://makandracards.com/makandra/24933-chrome-34+-firefox-38+-ie11+-ignore-autocomplete-off
mkhomedir_helper or create ssh homes with bash.rc and such
# warnings if some plugins are disabled, like make routes red
# replace show emails by https://docs.python.org/3/library/email.contentmanager.html#module-email.contentmanager
# setupforbiddendomains --url alexa -n 5000
* remove welcome box on dashboard?
# account contacts inline, show provided fields and ignore the rest?
# email usage -webkit-column-count:3;-moz-column-count:3;column-count:3;
# wordpressmu custom_url: set blog.domain
# validate_user on saas.wordpress to detect if username already exists before attempting to create a blog
# webapps don't override owner and permissions on every save(), just on create
# webapps php fpm allow pool config to be overriden. template + pool inheriting template?
# get_context signal to overridaconfiguration? best practice: all context on get_context, ever use other context. template rendering as backend generator: proof of concept
# DOmain show implicit records
# if not database_ready(): schedule a retry in 60 seconds, otherwise resources and other dynamic content gets fucked, maybe attach some 'signal' when first query goes trough
with database_ready:
shit_happend, otherwise schedule for first query
# Entry.objects.filter()[:1].first() (LIMIT 1)
# put "Coordinate Apache restart" inside a bash function for clarity
# mailscanner phishing, spam, whitelist choices
# show base and total desglosed
# CLOSE&DOWNLOAD doesn't redirect to anything, confusing for users
# Reverse lOgHistory order by date (lastest first)
* setuppostgres use porject_name for db name and user instead of orchestra
# POSTFIX web traffic monitor '": uid=" from=<%(user)s>'
# Mv .deleted make sure it works with nested destinations
# Automatically re-run backends until success? only timedout executions?
### Quick start
0. Install orchestra following any of these methods:
1. [PIP-only, Fast deployment setup (demo)](README.md#fast-deployment-setup)
2. [Docker container (development)](INSTALLDEV.md)
3. [Install on current system (production)](INSTALL.md)
1. Add the servers that you want to manage on `/admin/orchestration/servers` and copy orchestra's SSH pubkey to them
`orchestra@panel:~ ssh-copy-id root@server.address`
2. Now configure service by service (domains, databases, webapps, websites...):
1. Add the route through `/admin/orchestration/route/`
2. Check and configure related settings on `/admin/settings/setting/`
3. Configure related resources if needed `/resources/resource/`, like Account Disc limit and traffic.
3. Test that everything works by creating and deleting services
4. Do the same for the other services
3. Configure billing by adding services `/admin/services/service/add/` and plans `/admin/plans/plan/`
1. Once a service is created hit the *Update orders* button
### Architecture
Orchestration
Orders
### Creating new services
1. Think about if the service can fit into one of the existing models like: SaaS or WebApps, refere to the related documentation if that is the case.
2. Create a new django app using startapp management command. For ilustrational purposes we will create a crontab services that will allow orchestra to manage user-based crontabs.
`python3 manage.py startapp crontabs`
3. Add the new *crontabs* app to the `INSTALLED_APPS` in your project's `settings.py`
3. Create a `models.py` file with the data your service needs to keep in order to be managed by orchestra
```python
from django.db import models
class CrontabSchedule(models.Model):
account = models.ForeignKey('accounts.Account', verbose_name=_("account"))
minute = models.CharField(_("minute"), max_length=64, default='*')
hour = models.CharField(_("hour"), max_length=64, default='*')
day_of_week = models.CharField(_("day of week"), max_length=64, default='*')
day_of_month = models.CharField(_("day of month"), max_length=64, default='*')
month_of_year = models.CharField(_("month of year"), max_length=64, default='*')
class Meta:
ordering = ('month_of_year', 'day_of_month', 'day_of_week', 'hour', 'minute')
def __str__(self):
rfield = lambda f: f and str(f).replace(' ', '') or '*'
return "{0} {1} {2} {3} {4} (m/h/d/dM/MY)".format(
rfield(self.minute), rfield(self.hour), rfield(self.day_of_week),
rfield(self.day_of_month), rfield(self.month_of_year),
)
class Crontab(models.Model):
account = models.ForeignKey('accounts.Account', verbose_name=_("account"))
schedule = models.ForeignKey(CrontabSchedule, verbose_name=_("schedule"))
description = models.CharField(_("description"), max_length=256, blank=True)
command = models.TextField(_("content"))
def __str__(self):
return (self.description or self.command)[:32]
```
4. Create a `admin.py` to enable the admin interface
```python
from django.contrib import admin
from .models import CrontabSchedule, Crontab
class CrontabScheduleAdmin(admin.ModelAdmin):
pass
class CrontabAdmin(admin.ModelAdmin):
pass
admin.site.register(CrontabSchedule, CrontabScheduleAdmin)
admin.site.register(Crontab, CrontabAdmin)
5. Create a `api.py` to enable the REST API.
6. Create a `backends.py` fiel with the needed backends for service orchestration and monitoring
```python
import os
import textwrap
from django.utils.translation import ugettext_lazy as _
from orchestra.contrib.orchestration import ServiceController, replace
from orchestra.contrib.resources import ServiceMonitor
class UNIXCronBackend(ServiceController):
"""
Basic UNIX cron support.
"""
verbose_name = _("UNIX cron")
model = 'crons.CronTab'
def prepare(self):
super(UNIXCronBackend, self).prepare()
self.accounts = set()
def save(self, crontab):
self.accounts.add(crontab.account)
def delete(self, crontab):
self.accounts.add(crontab.account)
def commit(self):
for account in self.accounts:
crontab = None
self.append("echo '' > %(crontab_path)s" % context)
chown
for crontab in account.crontabs.all():
self.append("
# if crontab is None:
# self.append("rm -f %(crontab_path)s" % context)
```
7. Configure the routing