initial web tests

This commit is contained in:
Marc 2014-10-10 14:39:46 +00:00
parent b7758c97a5
commit 10e19fcdb4
26 changed files with 403 additions and 128 deletions

View File

@ -13,8 +13,8 @@ Note `*` _for sustancial progress_
2. [x] [Orchestra-orm](https://github.com/glic3rinu/orchestra-orm) a Python library for easily interacting with Orchestra REST API 2. [x] [Orchestra-orm](https://github.com/glic3rinu/orchestra-orm) a Python library for easily interacting with Orchestra REST API
3. [x] Service orchestration framework 3. [x] Service orchestration framework
4. [ ] Data model, crazy input validation, admin and REST interfaces, permissions, unit and functional tests, service management, migration scripts and documentation of: 4. [ ] Data model, crazy input validation, admin and REST interfaces, permissions, unit and functional tests, service management, migration scripts and documentation of:
1. [x] PHP/static Web applications 1. [ ] *PHP/static Web applications
1. [x] Websites with Apache 1. [ ] *Websites with Apache
2. [x] FTP/rsync/scp/shell system accounts 2. [x] FTP/rsync/scp/shell system accounts
2. [ ] *Databases and database users with MySQL 2. [ ] *Databases and database users with MySQL
1. [x] Mail accounts, aliases, forwards with Postfix and Dovecot 1. [x] Mail accounts, aliases, forwards with Postfix and Dovecot

42
TODO.md
View File

@ -16,9 +16,7 @@ TODO
* move invoice contact to invoices app? * move invoice contact to invoices app?
* PHPbBckendMiixin with get_php_ini * PHPbBckendMiixin with get_php_ini
* Apache: `IncludeOptional /etc/apache2/extra-vhos[t]/account-site-custom.con[f]` * Apache: `IncludeOptional /etc/apache2/extra-vhos[t]/account-site-custom.con[f]`
* rename account.user to main_user
* webmail identities and addresses * webmail identities and addresses
* cached -> cached_property
* user.roles.mailbox its awful when combined with addresses: * user.roles.mailbox its awful when combined with addresses:
* address.mailboxes filter by account is crap in admin and api * address.mailboxes filter by account is crap in admin and api
* address.mailboxes api needs a mailbox object endpoint (not nested user) * address.mailboxes api needs a mailbox object endpoint (not nested user)
@ -50,16 +48,13 @@ Remember that, as always with QuerySets, any subsequent chained methods which im
* EMAIL backend operations which contain stderr messages (because under certain failures status code is still 0) * EMAIL backend operations which contain stderr messages (because under certain failures status code is still 0)
* Settings dictionary like DRF2 in order to better override large settings like WEBSITES_APPLICATIONS.etc * Settings dictionary like DRF2 in order to better override large settings like WEBSITES_APPLICATIONS.etc
* DOCUMENT: orchestration.middleware: we need to know when an operation starts and ends in order to perform bulk server updates and also to wait for related objects to be saved (base object is saved first and then related) * DOCUMENT: orchestration.middleware: we need to know when an operation starts and ends in order to perform bulk server updates and also to wait for related objects to be saved (base object is saved first and then related)
orders.signales: we perform changes right away because data model state can change under monitoring and other periodik task, and we should keep orders consistency under any situation. orders.signales: we perform changes right away because data model state can change under monitoring and other periodik task, and we should keep orders consistency under any situation.
dependency collector with max_recursion that matches the number of dots on service.match and service.metric dependency collector with max_recursion that matches the number of dots on service.match and service.metric
* backend logs with hal logo * backend logs with hal logo
* Use logs for storing monitored values * Use logs for storing monitored values
* set_password orchestration method? * set_password orchestration method?
@ -94,9 +89,8 @@ Remember that, as always with QuerySets, any subsequent chained methods which im
return order.register_at.date() return order.register_at.date()
* mail backend related_models = ('resources__content_type') ?? * mail backend related_models = ('resources__content_type') ??
* ignore orders
* Dropdown menu for Account services/management object-tools * ignore orders (mark orders as ignored)
* Domain backend PowerDNS Bind validation support? * Domain backend PowerDNS Bind validation support?
@ -108,37 +102,23 @@ Remember that, as always with QuerySets, any subsequent chained methods which im
*jabber with mailbox accounts (dovecto mail notification) *jabber with mailbox accounts (dovecto mail notification)
* rename accounts register to manager register * rename accounts register to manager register or accounttools, accountutils
* make accounts django auth users
- when an account is created a mirrored system user is created
- system users are independent users, so they can have different passwords and all.
* take a look icons from ajenti ;) * take a look icons from ajenti ;)
* Disable services is_active should be computed on the fly in order to distinguish account.is_active from service.is_active when reactivation. * Disable services is_active should be computed on the fly in order to distinguish account.is_active from service.is_active when reactivation.
* Perhaps it is time to create a ServiceModel ? * Perhaps it is time to create a ServiceModel ?
* COpy account.main_user.username to account.name for performance
* service backend execution dependency? first create user on NIS master then create directories on service server
* prevent deletion of main user by the user itself * prevent deletion of main user by the user itself
* AccountAdminMixin auto adds 'account__name' on searchfields and handle account_link on fieldsets * AccountAdminMixin auto adds 'account__name' on searchfields and handle account_link on fieldsets
* Separate panel from server passwords? Store passwords on panel? set_password special backend operation? * Separate panel from server passwords? Store passwords on panel? set_password special backend operation?
* be more explicit about which backends are resources and which are service handling * be more explicit about which backends are resources and which are service handling
* What fields we really need on contacts? name email phone and what more? * What fields we really need on contacts? name email phone and what more?
* Redirect junk emails and delete every 30 days? * Redirect junk emails and delete every 30 days?
* DOC: Complitely decouples scripts execution, billing, service definition * DOC: Complitely decouples scripts execution, billing, service definition
@ -149,24 +129,12 @@ Remember that, as always with QuerySets, any subsequent chained methods which im
* Unify all users * Unify all users
* backend admin message with link
* delete main user -> delete account or prevent delete main user * delete main user -> delete account or prevent delete main user
APPS app?
* https://blog.flameeyes.eu/2011/01/mostly-unknown-openssh-tricks * https://blog.flameeyes.eu/2011/01/mostly-unknown-openssh-tricks
* Ansible orchestration *method* (methods.py) * Ansible orchestration *method* (methods.py)
* interdependency user <-> account with the old usermodel
* pip upgrade or install * pip upgrade or install
* disable account triggers save on cascade to execute backends save(update_field=[])
* validate database user names
* multiple domains creation; line separated domains * multiple domains creation; line separated domains
* Move MU webapps to SaaS?
* DN: Transaction atomicity and backend failure

View File

@ -49,6 +49,10 @@ class Account(auth.AbstractBaseUser):
def is_staff(self): def is_staff(self):
return self.is_superuser return self.is_superuser
@property
def main_systemuser(self):
return self.systemusers.get(is_main=True)
@classmethod @classmethod
def get_main(cls): def get_main(cls):
return cls.objects.get(pk=settings.ACCOUNTS_MAIN_PK) return cls.objects.get(pk=settings.ACCOUNTS_MAIN_PK)
@ -63,7 +67,8 @@ class Account(auth.AbstractBaseUser):
created = not self.pk created = not self.pk
super(Account, self).save(*args, **kwargs) super(Account, self).save(*args, **kwargs)
if created and hasattr(self, 'systemusers'): if created and hasattr(self, 'systemusers'):
self.systemusers.create_user(self.username, account=self, password=self.password, is_main=True) self.systemusers.create(username=self.username, account=self,
password=self.password, is_main=True)
def disable(self): def disable(self):
self.is_active = False self.is_active = False

View File

@ -15,6 +15,7 @@ class Bind9MasterDomainBackend(ServiceController):
('domains.Record', 'domain__origin'), ('domains.Record', 'domain__origin'),
('domains.Domain', 'origin'), ('domains.Domain', 'origin'),
) )
ignore_fields = ['serial']
@classmethod @classmethod
def is_main(cls, obj): def is_main(cls, obj):
@ -25,6 +26,7 @@ class Bind9MasterDomainBackend(ServiceController):
def save(self, domain): def save(self, domain):
context = self.get_context(domain) context = self.get_context(domain)
domain.refresh_serial() domain.refresh_serial()
print domain.render_zone()
context['zone'] = ';; %(banner)s\n' % context context['zone'] = ';; %(banner)s\n' % context
context['zone'] += domain.render_zone() context['zone'] += domain.render_zone()
self.append("{ echo -e '%(zone)s' | diff -N -I'^;;' %(zone_path)s - ; } ||" self.append("{ echo -e '%(zone)s' | diff -N -I'^;;' %(zone_path)s - ; } ||"

View File

@ -1,5 +1,4 @@
from django.db import models from django.db import models
from django.utils.functional import cached_property
from django.utils.translation import ugettext_lazy as _ from django.utils.translation import ugettext_lazy as _
from orchestra.core import services from orchestra.core import services

View File

@ -34,6 +34,8 @@ class DomainSerializer(AccountSerializerMixin, serializers.HyperlinkedModelSeria
try: try:
validators.validate_zone(domain.render_zone()) validators.validate_zone(domain.render_zone())
except ValidationError as err: except ValidationError as err:
self._errors = { 'all': err.message } self._errors = {
'all': err.message
}
return None return None
return instance return instance

View File

@ -8,7 +8,7 @@ from django.core.urlresolvers import reverse
from selenium.webdriver.support.select import Select from selenium.webdriver.support.select import Select
from orchestra.apps.orchestration.models import Server, Route from orchestra.apps.orchestration.models import Server, Route
from orchestra.utils.tests import BaseLiveServerTestCase, random_ascii, snapshot_on_error from orchestra.utils.tests import BaseLiveServerTestCase, random_ascii, snapshot_on_error, save_response_on_error
from orchestra.utils.system import run from orchestra.utils.system import run
from ... import settings, utils, backends from ... import settings, utils, backends
@ -129,7 +129,7 @@ class DomainTestMixin(object):
'domain_name': domain_name, 'domain_name': domain_name,
'server_addr': server_addr 'server_addr': server_addr
} }
dig_soa = 'dig @%(server_addr)s %(domain_name)s SOA|grep "\sSOA\s"' dig_soa = 'dig @%(server_addr)s %(domain_name)s SOA | grep "\sSOA\s"'
soa = run(dig_soa % context).stdout.split() soa = run(dig_soa % context).stdout.split()
# testdomain.org. 3600 IN SOA ns.example.com. hostmaster.example.com. 2014021100 86400 7200 2419200 3600 # testdomain.org. 3600 IN SOA ns.example.com. hostmaster.example.com. 2014021100 86400 7200 2419200 3600
self.assertEqual('%(domain_name)s.' % context, soa[0]) self.assertEqual('%(domain_name)s.' % context, soa[0])
@ -140,7 +140,7 @@ class DomainTestMixin(object):
hostmaster = utils.format_hostmaster(settings.DOMAINS_DEFAULT_HOSTMASTER) hostmaster = utils.format_hostmaster(settings.DOMAINS_DEFAULT_HOSTMASTER)
self.assertEqual(hostmaster, soa[5]) self.assertEqual(hostmaster, soa[5])
dig_ns = 'dig @%(server_addr)s %(domain_name)s NS|grep "\sNS\s"' dig_ns = 'dig @%(server_addr)s %(domain_name)s NS |grep "\sNS\s"'
name_servers = run(dig_ns % context).stdout name_servers = run(dig_ns % context).stdout
ns_records = ['ns1.%s.' % self.domain_name, 'ns2.%s.' % self.domain_name] ns_records = ['ns1.%s.' % self.domain_name, 'ns2.%s.' % self.domain_name]
self.assertEqual(2, len(name_servers.splitlines())) self.assertEqual(2, len(name_servers.splitlines()))
@ -153,7 +153,7 @@ class DomainTestMixin(object):
self.assertEqual('NS', ns[3]) self.assertEqual('NS', ns[3])
self.assertIn(ns[4], ns_records) self.assertIn(ns[4], ns_records)
dig_mx = 'dig @%(server_addr)s %(domain_name)s MX|grep "\sMX\s"' dig_mx = 'dig @%(server_addr)s %(domain_name)s MX | grep "\sMX\s"'
mx = run(dig_mx % context).stdout.split() mx = run(dig_mx % context).stdout.split()
# testdomain.org. 3600 IN MX 10 orchestra.lan. # testdomain.org. 3600 IN MX 10 orchestra.lan.
self.assertEqual('%(domain_name)s.' % context, mx[0]) self.assertEqual('%(domain_name)s.' % context, mx[0])
@ -163,7 +163,7 @@ class DomainTestMixin(object):
self.assertIn(mx[4], ['30', '40']) self.assertIn(mx[4], ['30', '40'])
self.assertIn(mx[5], ['mail3.orchestra.lan.', 'mail4.orchestra.lan.']) self.assertIn(mx[5], ['mail3.orchestra.lan.', 'mail4.orchestra.lan.'])
dig_cname = 'dig @%(server_addr)s www.%(domain_name)s CNAME|grep "\sCNAME\s"' dig_cname = 'dig @%(server_addr)s www.%(domain_name)s CNAME | grep "\sCNAME\s"'
cname = run(dig_cname % context).stdout.split() cname = run(dig_cname % context).stdout.split()
# testdomain.org. 3600 IN MX 10 orchestra.lan. # testdomain.org. 3600 IN MX 10 orchestra.lan.
self.assertEqual('www.%(domain_name)s.' % context, cname[0]) self.assertEqual('www.%(domain_name)s.' % context, cname[0])
@ -194,13 +194,13 @@ class DomainTestMixin(object):
self.add(self.ns1_name, self.ns1_records) self.add(self.ns1_name, self.ns1_records)
self.add(self.ns2_name, self.ns2_records) self.add(self.ns2_name, self.ns2_records)
self.add(self.domain_name, self.domain_records) self.add(self.domain_name, self.domain_records)
self.addCleanup(partial(self.delete, self.domain_name)) # self.addCleanup(partial(self.delete, self.domain_name))
self.update(self.domain_name, self.domain_update_records) self.update(self.domain_name, self.domain_update_records)
self.add(self.www_name, self.www_records) # self.add(self.www_name, self.www_records)
time.sleep(0.5) time.sleep(0.5)
self.validate_update(self.MASTER_SERVER_ADDR, self.domain_name) self.validate_update(self.MASTER_SERVER_ADDR, self.domain_name)
time.sleep(5) # time.sleep(5)
self.validate_update(self.SLAVE_SERVER_ADDR, self.domain_name) # self.validate_update(self.SLAVE_SERVER_ADDR, self.domain_name)
def test_add_add_delete_delete(self): def test_add_add_delete_delete(self):
self.add(self.ns1_name, self.ns1_records) self.add(self.ns1_name, self.ns1_records)
@ -276,15 +276,18 @@ class RESTDomainMixin(DomainTestMixin):
self.rest_login() self.rest_login()
self.add_route() self.add_route()
@save_response_on_error
def add(self, domain_name, records): def add(self, domain_name, records):
records = [ dict(type=type, value=value) for type,value in records ] records = [ dict(type=type, value=value) for type,value in records ]
self.rest.domains.create(name=domain_name, records=records) self.rest.domains.create(name=domain_name, records=records)
@save_response_on_error
def delete(self, domain_name): def delete(self, domain_name):
domain = Domain.objects.get(name=domain_name) domain = Domain.objects.get(name=domain_name)
domain = self.rest.domains.retrieve(id=domain.pk) domain = self.rest.domains.retrieve(id=domain.pk)
domain.delete() domain.delete()
@save_response_on_error
def update(self, domain_name, records): def update(self, domain_name, records):
records = [ dict(type=type, value=value) for type,value in records ] records = [ dict(type=type, value=value) for type,value in records ]
domains = self.rest.domains.retrieve(name=domain_name) domains = self.rest.domains.retrieve(name=domain_name)

View File

@ -64,7 +64,7 @@ class ListMixin(object):
backend = backends.MailmanBackend.get_name() backend = backends.MailmanBackend.get_name()
Route.objects.create(backend=backend, match=True, host=server) Route.objects.create(backend=backend, match=True, host=server)
def atest_add(self): def test_add(self):
name = '%s_list' % random_ascii(10) name = '%s_list' % random_ascii(10)
password = '@!?%spppP001' % random_ascii(5) password = '@!?%spppP001' % random_ascii(5)
admin_email = 'root@test3.orchestra.lan' admin_email = 'root@test3.orchestra.lan'
@ -100,8 +100,8 @@ class RESTListMixin(ListMixin):
self.rest.lists.create(name=name, password=password, admin_email=admin_email, **extra) self.rest.lists.create(name=name, password=password, admin_email=admin_email, **extra)
@save_response_on_error @save_response_on_error
def delete(self, username): def delete(self, name):
list = self.rest.lists.retrieve(name=username).get() list = self.rest.lists.retrieve(name=name).get()
list.delete() list.delete()

View File

@ -14,6 +14,7 @@ logger = logging.getLogger(__name__)
def as_task(execute): def as_task(execute):
def wrapper(*args, **kwargs): def wrapper(*args, **kwargs):
""" failures on the backend execution doesn't fuck the request transaction atomicity """
db.transaction.set_autocommit(False) db.transaction.set_autocommit(False)
try: try:
log = execute(*args, **kwargs) log = execute(*args, **kwargs)

View File

@ -49,6 +49,7 @@ class OperationsMiddleware(object):
request = getattr(cls.thread_locals, 'request', None) request = getattr(cls.thread_locals, 'request', None)
if request is None: if request is None:
return return
good_action = action
pending_operations = cls.get_pending_operations() pending_operations = cls.get_pending_operations()
for backend in ServiceBackend.get_backends(): for backend in ServiceBackend.get_backends():
instance = None instance = None
@ -75,7 +76,7 @@ class OperationsMiddleware(object):
if update_fields: if update_fields:
# "update_fileds=[]" is a convention for explicitly executing backend # "update_fileds=[]" is a convention for explicitly executing backend
# i.e. account.disable() # i.e. account.disable()
if not update_fields == []: if update_fields != []:
execute = False execute = False
for field in update_fields: for field in update_fields:
if field not in backend.ignore_fields: if field not in backend.ignore_fields:
@ -84,12 +85,17 @@ class OperationsMiddleware(object):
if not execute: if not execute:
continue continue
instance = copy.copy(instance) instance = copy.copy(instance)
good = instance
operation = Operation.create(backend, instance, action) operation = Operation.create(backend, instance, action)
if action != Operation.DELETE: if action != Operation.DELETE:
# usually we expect to be using last object state, # usually we expect to be using last object state,
# except when we are deleting it # except when we are deleting it
pending_operations.discard(operation) pending_operations.discard(operation)
pending_operations.add(operation) pending_operations.add(operation)
try:
print kwargs['instance'], good_action
except:
pass
def process_request(self, request): def process_request(self, request):
""" Store request on a thread local variable """ """ Store request on a thread local variable """

View File

@ -82,9 +82,9 @@ class SystemUserMixin(object):
# Home will be deleted on account delete, see test_delete_account # Home will be deleted on account delete, see test_delete_account
def validate_ftp(self, username, password): def validate_ftp(self, username, password):
connection = ftplib.FTP(self.MASTER_SERVER) ftp = ftplib.FTP(self.MASTER_SERVER)
connection.login(user=username, passwd=password) ftp.login(user=username, passwd=password)
connection.close() ftp.close()
def validate_sftp(self, username, password): def validate_sftp(self, username, password):
transport = paramiko.Transport((self.MASTER_SERVER, 22)) transport = paramiko.Transport((self.MASTER_SERVER, 22))

View File

@ -1,23 +1,31 @@
import pkgutil import pkgutil
import textwrap
class WebAppServiceMixin(object): class WebAppServiceMixin(object):
model = 'webapps.WebApp' model = 'webapps.WebApp'
def create_webapp_dir(self, context): def create_webapp_dir(self, context):
self.append("mkdir -p '%(app_path)s'" % context) self.append(textwrap.dedent("""
self.append("chown %(user)s.%(group)s '%(app_path)s'" % context) path=""
for dir in $(echo %(app_path)s | tr "/" "\n"); do
path="${path}/${dir}"
[ -d $path ] || {
mkdir "${path}"
chown %(user)s.%(group)s "${path}"
}
done
""" % context))
def delete_webapp_dir(self, context): def delete_webapp_dir(self, context):
self.append("rm -fr %(app_path)s" % context) self.append("rm -fr %(app_path)s" % context)
def get_context(self, webapp): def get_context(self, webapp):
return { return {
'user': webapp.account.user.username, 'user': webapp.account.username,
'group': webapp.account.user.username, 'group': webapp.account.username,
'app_name': webapp.name, 'app_name': webapp.name,
'type': webapp.type, 'type': webapp.type,
'app_path': webapp.get_path(), 'app_path': webapp.get_path().rstrip('/'),
'banner': self.get_banner(), 'banner': self.get_banner(),
} }

View File

@ -1,9 +0,0 @@
from django.utils.translation import ugettext_lazy as _
from orchestra.apps.orchestration import ServiceController
from . import WebAppServiceMixin
class AwstatsBackend(WebAppServiceMixin, ServiceController):
verbose_name = _("Awstats")

View File

@ -1,4 +1,5 @@
import os import os
import textwrap
from django.utils.translation import ugettext_lazy as _ from django.utils.translation import ugettext_lazy as _
@ -15,9 +16,12 @@ class PHPFcgidBackend(WebAppServiceMixin, ServiceController):
context = self.get_context(webapp) context = self.get_context(webapp)
self.create_webapp_dir(context) self.create_webapp_dir(context)
self.append("mkdir -p %(wrapper_dir)s" % context) self.append("mkdir -p %(wrapper_dir)s" % context)
self.append( self.append(textwrap.dedent("""\
"{ echo -e '%(wrapper_content)s' | diff -N -I'^\s*#' %(wrapper_path)s - ; } ||" {
" { echo -e '%(wrapper_content)s' > %(wrapper_path)s; UPDATED=1; }" % context) echo -e '%(wrapper_content)s' | diff -N -I'^\s*#' %(wrapper_path)s -
} || {
echo -e '%(wrapper_content)s' > %(wrapper_path)s; UPDATED_APACHE=1
}""" % context))
self.append("chmod +x %(wrapper_path)s" % context) self.append("chmod +x %(wrapper_path)s" % context)
self.append("chown -R %(user)s.%(group)s %(wrapper_dir)s" % context) self.append("chown -R %(user)s.%(group)s %(wrapper_dir)s" % context)
@ -25,6 +29,10 @@ class PHPFcgidBackend(WebAppServiceMixin, ServiceController):
context = self.get_context(webapp) context = self.get_context(webapp)
self.delete_webapp_dir(context) self.delete_webapp_dir(context)
def commit(self):
super(PHPFcgidBackend, self).commit()
self.append("[[ $UPDATED_APACHE == 1 ]] && { /etc/init.d/apache reload; }")
def get_context(self, webapp): def get_context(self, webapp):
context = super(PHPFcgidBackend, self).get_context(webapp) context = super(PHPFcgidBackend, self).get_context(webapp)
init_vars = webapp.get_php_init_vars() init_vars = webapp.get_php_init_vars()
@ -36,12 +44,12 @@ class PHPFcgidBackend(WebAppServiceMixin, ServiceController):
context['init_vars'] = '' context['init_vars'] = ''
wrapper_path = settings.WEBAPPS_FCGID_PATH % context wrapper_path = settings.WEBAPPS_FCGID_PATH % context
context.update({ context.update({
'wrapper_content': ( 'wrapper_content': textwrap.dedent("""\
"#!/bin/sh\n" #!/bin/sh
"# %(banner)s\n" # %(banner)s
"export PHPRC=/etc/%(type)s/cgi/\n" export PHPRC=/etc/%(type)s/cgi/
"exec /usr/bin/%(type)s-cgi %(init_vars)s\n" exec /usr/bin/%(type)s-cgi %(init_vars)s
) % context, """ % context),
'wrapper_path': wrapper_path, 'wrapper_path': wrapper_path,
'wrapper_dir': os.path.dirname(wrapper_path), 'wrapper_dir': os.path.dirname(wrapper_path),
}) })

View File

@ -36,8 +36,8 @@ class PHPFPMBackend(WebAppServiceMixin, ServiceController):
}) })
context['fpm_listen'] = settings.WEBAPPS_FPM_LISTEN % context context['fpm_listen'] = settings.WEBAPPS_FPM_LISTEN % context
fpm_config = Template( fpm_config = Template(
"[{{ user }}]\n"
";; {{ banner }}\n" ";; {{ banner }}\n"
"[{{ user }}]\n"
"user = {{ user }}\n" "user = {{ user }}\n"
"group = {{ group }}\n\n" "group = {{ group }}\n\n"
"listen = {{ fpm_listen | safe }}\n" "listen = {{ fpm_listen | safe }}\n"

View File

@ -57,7 +57,7 @@ class WebApp(models.Model):
return init_vars return init_vars
def get_fpm_port(self): def get_fpm_port(self):
return settings.WEBAPPS_FPM_START_PORT + self.account.user.pk return settings.WEBAPPS_FPM_START_PORT + self.account.pk
def get_method(self): def get_method(self):
method = settings.WEBAPPS_TYPES[self.type] method = settings.WEBAPPS_TYPES[self.type]
@ -66,7 +66,7 @@ class WebApp(models.Model):
def get_path(self): def get_path(self):
context = { context = {
'user': self.account.user, 'user': self.account.username,
'app_name': self.name, 'app_name': self.name,
} }
return settings.WEBAPPS_BASE_ROOT % context return settings.WEBAPPS_BASE_ROOT % context

View File

View File

@ -0,0 +1,181 @@
import ftplib
import os
import time
import textwrap
from StringIO import StringIO
from django.conf import settings as djsettings
from django.contrib.contenttypes.models import ContentType
from django.core.management.base import CommandError
from django.core.urlresolvers import reverse
from selenium.webdriver.support.select import Select
from orchestra.apps.accounts.models import Account
from orchestra.apps.domains.models import Domain
from orchestra.apps.orchestration.models import Server, Route
from orchestra.apps.resources.models import Resource
from orchestra.apps.systemusers.backends import SystemUserBackend
from orchestra.utils.system import run, sshrun
from orchestra.utils.tests import BaseLiveServerTestCase, random_ascii, snapshot_on_error, save_response_on_error
from ... import backends, settings
from ...models import WebApp
class WebAppMixin(object):
MASTER_SERVER = os.environ.get('ORCHESTRA_MASTER_SERVER', 'localhost')
DEPENDENCIES = (
'orchestra.apps.orchestration',
'orchestra.apps.systemusers',
'orchestra.apps.webapps',
)
def setUp(self):
super(WebAppMixin, self).setUp()
self.add_route()
djsettings.DEBUG = True
def add_route(self):
# backends = [
# # TODO MU apps on SaaS?
# backends.awstats.AwstatsBackend,
# backends.dokuwikimu.DokuWikiMuBackend,
# backends.drupalmu.DrupalMuBackend,
# backends.phpfcgid.PHPFcgidBackend,
# backends.phpfpm.PHPFPMBackend,
# backends.static.StaticBackend,
# backends.wordpressmu.WordpressMuBackend,
# ]
server = Server.objects.create(name=self.MASTER_SERVER)
for backend in [SystemUserBackend, self.backend]:
backend = backend.get_name()
Route.objects.create(backend=backend, match=True, host=server)
def test_add(self):
name = '%s_%s_webapp' % (random_ascii(10), self.type_value)
self.add_webapp(name)
self.validate_add_webapp(name)
# self.addCleanup(self.delete, username)
class StaticWebAppMixin(object):
backend = backends.static.StaticBackend
type_value = 'static'
token = random_ascii(100)
page = (
'index.html',
'<html>Hello World! %s </html>\n' % token,
'<html>Hello World! %s </html>\n' % token,
)
def validate_add_webapp(self, name):
try:
ftp = ftplib.FTP(self.MASTER_SERVER)
ftp.login(user=self.account.username, passwd=self.account_password)
ftp.cwd('webapps/%s' % name)
index = StringIO()
index.write(self.page[1])
index.seek(0)
ftp.storbinary('STOR %s' % self.page[0], index)
index.close()
finally:
ftp.close()
class PHPFcidWebAppMixin(StaticWebAppMixin):
backend = backends.phpfcgid.PHPFcgidBackend
type_value = 'php5'
token = random_ascii(100)
page = (
'index.php',
'<?php print("Hello World! %s");\n?>\n' % token,
'Hello World! %s' % token,
)
class PHPFPMWebAppMixin(StaticWebAppMixin):
backend = backends.phpfpm.PHPFPMBackend
type_value = 'php5.5'
token = random_ascii(100)
page = (
'index.php',
'<?php print("Hello World! %s");\n?>\n' % token,
'Hello World! %s' % token,
)
class RESTWebAppMixin(object):
def setUp(self):
super(RESTWebAppMixin, self).setUp()
self.rest_login()
# create main user
self.save_systemuser()
@save_response_on_error
def save_systemuser(self):
self.rest.systemusers.retrieve().get().save()
@save_response_on_error
def add_webapp(self, name, options=[]):
self.rest.webapps.create(name=name, type=self.type_value)
@save_response_on_error
def delete_webapp(self, name):
list = self.rest.lists.retrieve(name=name).get()
list.delete()
class AdminWebAppMixin(WebAppMixin):
def setUp(self):
super(AdminWebAppMixin, self).setUp()
self.admin_login()
# create main user
self.save_systemuser()
# TODO save_account()
@snapshot_on_error
def add(self, name, password, admin_email):
url = self.live_server_url + reverse('admin:mails_List_add')
self.selenium.get(url)
account_input = self.selenium.find_element_by_id('id_account')
account_select = Select(account_input)
account_select.select_by_value(str(self.account.pk))
name_field = self.selenium.find_element_by_id('id_name')
name_field.send_keys(username)
password_field = self.selenium.find_element_by_id('id_password1')
password_field.send_keys(password)
password_field = self.selenium.find_element_by_id('id_password2')
password_field.send_keys(password)
if quota is not None:
quota_id = 'id_resources-resourcedata-content_type-object_id-0-allocated'
quota_field = self.selenium.find_element_by_id(quota_id)
quota_field.clear()
quota_field.send_keys(quota)
if filtering is not None:
filtering_input = self.selenium.find_element_by_id('id_filtering')
filtering_select = Select(filtering_input)
filtering_select.select_by_value("CUSTOM")
filtering_inline = self.selenium.find_element_by_id('fieldsetcollapser0')
filtering_inline.click()
time.sleep(0.5)
filtering_field = self.selenium.find_element_by_id('id_custom_filtering')
filtering_field.send_keys(filtering)
name_field.submit()
self.assertNotEqual(url, self.selenium.current_url)
class RESTWebAppTest(PHPFcidWebAppMixin, RESTWebAppMixin, WebAppMixin, BaseLiveServerTestCase):
pass
#class AdminWebAppTest(AdminWebAppMixin, BaseLiveServerTestCase):
# pass

View File

@ -30,15 +30,13 @@ class Apache2Backend(ServiceController):
apache_conf = Template(textwrap.dedent("""\ apache_conf = Template(textwrap.dedent("""\
# {{ banner }} # {{ banner }}
<VirtualHost *:{{ site.port }}> <VirtualHost *:{{ site.port }}>
ServerName {{ site.domains.all|first }} ServerName {{ site.domains.all|first }}\
{% if site.domains.all|slice:"1:" %} {% if site.domains.all|slice:"1:" %}
ServerAlias {{ site.domains.all|slice:"1:"|join:' ' }} ServerAlias {{ site.domains.all|slice:"1:"|join:' ' }}{% endif %}
{% endif %}
CustomLog {{ logs }} common CustomLog {{ logs }} common
SuexecUserGroup {{ user }} {{ group }} SuexecUserGroup {{ user }} {{ group }}\
{% for line in extra_conf.splitlines %}" {% for line in extra_conf.splitlines %}
{{ line | safe }} {{ line | safe }}{% endfor %}
{% endfor %}
</VirtualHost>""" </VirtualHost>"""
)) ))
apache_conf = apache_conf.render(Context(context)) apache_conf = apache_conf.render(Context(context))
@ -83,13 +81,13 @@ class Apache2Backend(ServiceController):
context = self.get_content_context(content) context = self.get_content_context(content)
context['fcgid_path'] = fcgid_path % context context['fcgid_path'] = fcgid_path % context
fcgid = self.get_alias_directives(content) fcgid = self.get_alias_directives(content)
fcgid += ( fcgid += textwrap.dedent("""\
"ProxyPass %(location)s !\n" ProxyPass %(location)s !
"<Directory %(app_path)s>\n" <Directory %(app_path)s>
" Options +ExecCGI\n" Options +ExecCGI
" AddHandler fcgid-script .php\n" AddHandler fcgid-script .php
" FcgidWrapper %(fcgid_path)s\n" FcgidWrapper %(fcgid_path)s
) % context """ % context)
for option in content.webapp.options.filter(name__startswith='Fcgid'): for option in content.webapp.options.filter(name__startswith='Fcgid'):
fcgid += " %s %s\n" % (option.name, option.value) fcgid += " %s %s\n" % (option.name, option.value)
fcgid += "</Directory>\n" fcgid += "</Directory>\n"
@ -100,11 +98,11 @@ class Apache2Backend(ServiceController):
custom_cert = site.options.filter(name='ssl') custom_cert = site.options.filter(name='ssl')
if custom_cert: if custom_cert:
cert = tuple(custom_cert[0].value.split()) cert = tuple(custom_cert[0].value.split())
directives = ( directives = textwrap.dedent("""\
"SSLEngine on\n" SSLEngine on
"SSLCertificateFile %s\n" SSLCertificateFile %s
"SSLCertificateKeyFile %s\n" SSLCertificateKeyFile %s""" % cert
) % cert )
return directives return directives
def get_security(self, site): def get_security(self, site):
@ -129,17 +127,17 @@ class Apache2Backend(ServiceController):
path, name, passwd = re.match(regex, protection.value).groups() path, name, passwd = re.match(regex, protection.value).groups()
path = os.path.join(context['root'], path) path = os.path.join(context['root'], path)
passwd = os.path.join(self.USER_HOME % context, passwd) passwd = os.path.join(self.USER_HOME % context, passwd)
protections += ("\n" protections += textwrap.dedent("""
"<Directory %s>\n" <Directory %s>
" AllowOverride All\n" AllowOverride All
# " AuthPAM_Enabled off\n" #AuthPAM_Enabled off
" AuthType Basic\n" AuthType Basic
" AuthName %s\n" AuthName %s
" AuthUserFile %s\n" AuthUserFile %s
" <Limit GET POST>\n" <Limit GET POST>
" require valid-user\n" require valid-user
" </Limit>\n" </Limit>
"</Directory>\n" % (path, name, passwd) </Directory>""" % (path, name, passwd)
) )
return protections return protections
@ -161,8 +159,8 @@ class Apache2Backend(ServiceController):
'site': site, 'site': site,
'site_name': site.name, 'site_name': site.name,
'site_unique_name': site.unique_name, 'site_unique_name': site.unique_name,
'user': site.account.user.username, 'user': site.account.username,
'group': site.account.user.username, 'group': site.account.username,
'sites_enabled': sites_enabled, 'sites_enabled': sites_enabled,
'sites_available': "%s.conf" % os.path.join(sites_available, site.unique_name), 'sites_available': "%s.conf" % os.path.join(sites_available, site.unique_name),
'logs': os.path.join(settings.WEBSITES_BASE_APACHE_LOGS, site.unique_name), 'logs': os.path.join(settings.WEBSITES_BASE_APACHE_LOGS, site.unique_name),
@ -190,7 +188,7 @@ class Apache2Traffic(ServiceMonitor):
def prepare(self): def prepare(self):
current_date = timezone.localtime(self.current_date) current_date = timezone.localtime(self.current_date)
current_date = current_date.strftime("%Y%m%d%H%M%S") current_date = current_date.strftime("%Y%m%d%H%M%S")
self.append(textwrap.dedent(""" self.append(textwrap.dedent("""\
function monitor () { function monitor () {
OBJECT_ID=$1 OBJECT_ID=$1
INI_DATE=$2 INI_DATE=$2

View File

@ -0,0 +1,91 @@
import os
import socket
import time
import textwrap
from django.conf import settings as djsettings
from django.contrib.contenttypes.models import ContentType
from django.core.management.base import CommandError
from django.core.urlresolvers import reverse
import requests
from selenium.webdriver.support.select import Select
from orchestra.apps.accounts.models import Account
from orchestra.apps.domains.models import Domain, Record
from orchestra.apps.domains.backends import Bind9MasterDomainBackend
from orchestra.apps.orchestration.models import Server, Route
from orchestra.apps.resources.models import Resource
from orchestra.apps.webapps.tests.functional_tests.tests import StaticWebAppMixin, RESTWebAppMixin, WebAppMixin, PHPFcidWebAppMixin, PHPFPMWebAppMixin
from orchestra.utils.system import run, sshrun
from orchestra.utils.tests import BaseLiveServerTestCase, random_ascii, snapshot_on_error, save_response_on_error
from ... import backends, settings
from ...models import Website
class WebsiteMixin(WebAppMixin):
MASTER_SERVER = os.environ.get('ORCHESTRA_MASTER_SERVER', 'localhost')
MASTER_SERVER_ADDR = socket.gethostbyname(MASTER_SERVER)
DEPENDENCIES = (
'orchestra.apps.orchestration',
'orchestra.apps.domains',
'orchestra.apps.websites',
'orchestra.apps.webapps',
'orchestra.apps.systemusers',
)
def add_route(self):
super(WebsiteMixin, self).add_route()
server = Server.objects.get()
backend = backends.apache.Apache2Backend.get_name()
Route.objects.create(backend=backend, match=True, host=server)
backend = Bind9MasterDomainBackend.get_name()
Route.objects.create(backend=backend, match=True, host=server)
def validate_add_website(self, name, domain):
url = 'http://%s/%s' % (domain.name, self.page[0])
self.assertEqual(self.page[2], requests.get(url).content)
def test_add(self):
# TODO domains with "_" bad name!
domain_name = '%sdomain.lan' % random_ascii(10)
domain = Domain.objects.create(name=domain_name, account=self.account)
domain.records.create(type=Record.A, value=self.MASTER_SERVER_ADDR)
self.save_domain(domain)
webapp = '%s_%s_webapp' % (random_ascii(10), self.type_value)
self.add_webapp(webapp)
self.validate_add_webapp(webapp)
website = '%s_website' % random_ascii(10)
self.add_website(website, domain, webapp)
self.validate_add_website(website, domain)
class RESTWebsiteMixin(RESTWebAppMixin):
@save_response_on_error
def save_domain(self, domain):
self.rest.domains.retrieve().get().save()
def add_website(self, name, domain, webapp):
domain = self.rest.domains.retrieve().get()
webapp = self.rest.webapps.retrieve().get()
self.rest.websites.create(name=name, domains=[domain.url], contents=[{'webapp': webapp.url}])
#class RESTWebsiteTest(RESTWebsiteMixin, StaticWebAppMixin, WebsiteMixin, BaseLiveServerTestCase):
# pass
PHPFPMWebAppMixin
#class RESTWebsiteTest(RESTWebsiteMixin, PHPFcidWebAppMixin, WebsiteMixin, BaseLiveServerTestCase):
# pass
class RESTWebsiteTest(RESTWebsiteMixin, PHPFPMWebAppMixin, WebsiteMixin, BaseLiveServerTestCase):
pass
#class AdminWebsiteTest(AdminWebsiteMixin, BaseLiveServerTestCase):
# pass

View File

@ -41,7 +41,7 @@ def validate_name(value):
""" """
A single non-empty line of free-form text with no whitespace. A single non-empty line of free-form text with no whitespace.
""" """
validators.RegexValidator('^\w+$', validators.RegexValidator('^[\.\w]+$',
_("Enter a valid name (text without whitspaces)."), 'invalid')(value) _("Enter a valid name (text without whitspaces)."), 'invalid')(value)

View File

@ -28,12 +28,14 @@ The goal of this setup is having a high-performance state-of-the-art deployment
apt-get install apache2-mpm-event php5-fpm libapache2-mod-fcgid apache2-suexec-custom php5-cgi apt-get install apache2-mpm-event php5-fpm libapache2-mod-fcgid apache2-suexec-custom php5-cgi
``` ```
# TODO libapache2-mod-auth-pam is no longer part of the debian distribution,
# replace with libapache2-mod-authnz-external pwauth
2. Enable some convinient Apache modules 2. Enable some convinient Apache modules
```bash ```bash
a2enmod suexec a2enmod suexec
a2enmod ssl a2enmod ssl
a2enmod auth_pam #a2enmod auth_pam
a2enmod proxy_fcgi a2enmod proxy_fcgi
a2emmod userdir a2emmod userdir
``` ```
@ -55,15 +57,18 @@ The goal of this setup is having a high-performance state-of-the-art deployment
``` ```
5. Restart Apache 5. Restart Apache
```bash ```bash
service apache2 restart service apache2 restart
``` ```
* TODO
libapache2-mod-auth-pam
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=710770
* ExecCGI * ExecCGI
@ -73,6 +78,11 @@ The goal of this setup is having a high-performance state-of-the-art deployment
</Directory> </Directory>
``` ```
* Permissions
<Directory /home/*/webapps>
Require all granted
</Directory>
TODO CHRoot TODO CHRoot
https://andrewbevitt.com/tutorials/apache-varnish-chrooted-php-fpm-wordpress-virtual-host/ https://andrewbevitt.com/tutorials/apache-varnish-chrooted-php-fpm-wordpress-virtual-host/

View File

@ -10,10 +10,12 @@ VsFTPd with System Users
2. Make some configurations 2. Make some configurations
```bash ```bash
sed -i "s/anonymous_enable=YES/anonymous_enable=NO/" /etc/vsftpd.conf sed -i "s/^anonymous_enable=YES/anonymous_enable=NO/" /etc/vsftpd.conf
sed -i "s/#local_enable=YES/local_enable=YES/" /etc/vsftpd.conf sed -i "s/^#local_enable=YES/local_enable=YES/" /etc/vsftpd.conf
sed -i "s/#write_enable=YES/write_enable=YES/" /etc/vsftpd.conf sed -i "s/^#write_enable=YES/write_enable=YES/" /etc/vsftpd.conf
# sed -i "s/#chroot_local_user=YES/chroot_local_user=YES/" /etc/vsftpd.conf # sed -i "s/^#chroot_local_user=YES/chroot_local_user=YES/" /etc/vsftpd.conf
sed -i "s/^#local_umask=022/local_umask=022/" /etc/vsftpd.conf
echo '/dev/null' >> /etc/shells echo '/dev/null' >> /etc/shells
``` ```