initial web tests

ROADMAP.md

@@ -13,8 +13,8 @@ Note `*` _for sustancial progress_
 2. [x] [Orchestra-orm](https://github.com/glic3rinu/orchestra-orm) a Python library for easily interacting with Orchestra REST API
 3. [x] Service orchestration framework
 4. [ ] Data model, crazy input validation, admin and REST interfaces, permissions, unit and functional tests, service management, migration scripts and documentation of:
-    1. [x] PHP/static Web applications
+    1. [ ] *PHP/static Web applications
-    1. [x] Websites with Apache
+    1. [ ] *Websites with Apache
     2. [x] FTP/rsync/scp/shell system accounts
     2. [ ] *Databases and database users with MySQL
     1. [x] Mail accounts, aliases, forwards with Postfix and Dovecot

TODO.md

@@ -16,9 +16,7 @@ TODO
 * move invoice contact to invoices app?
 * PHPbBckendMiixin with get_php_ini
 * Apache: `IncludeOptional /etc/apache2/extra-vhos[t]/account-site-custom.con[f]`
-* rename account.user to main_user
 * webmail identities and addresses
-* cached -> cached_property
 * user.roles.mailbox its awful when combined with addresses:
     * address.mailboxes filter by account is crap in admin and api
     * address.mailboxes api needs a mailbox object endpoint (not nested user)
@@ -50,16 +48,13 @@ Remember that, as always with QuerySets, any subsequent chained methods which im
 
 * EMAIL backend operations which contain stderr messages (because under certain failures status code is still 0)
 
-
 * Settings dictionary like DRF2 in order to better override large settings like WEBSITES_APPLICATIONS.etc
 
-
 * DOCUMENT: orchestration.middleware: we need to know when an operation starts and ends in order to perform bulk server updates and also to wait for related objects to be saved (base object is saved first and then related)
             orders.signales: we perform changes right away because data model state can change under monitoring and other periodik task, and we should keep orders consistency under any situation.
                              dependency collector with max_recursion that matches the number of dots on service.match and service.metric
 
 
-
 * backend logs with hal logo
 * Use logs for storing monitored values
 * set_password orchestration method?
@@ -94,9 +89,8 @@ Remember that, as always with QuerySets, any subsequent chained methods which im
         return order.register_at.date()
 
 * mail backend related_models = ('resources__content_type') ??
-* ignore orders
 
-* Dropdown menu for Account services/management object-tools
+* ignore orders (mark orders as ignored)
 
 * Domain backend PowerDNS Bind validation support?
 
@@ -108,37 +102,23 @@ Remember that, as always with QuerySets, any subsequent chained methods which im
 
 *jabber with mailbox accounts (dovecto mail notification)
 
-* rename accounts register to manager register
+* rename accounts register to manager register or accounttools, accountutils
-
-* make accounts django auth users
-    - when an account is created a mirrored system user is created
-    - system users are independent users, so they can have different passwords and all.
 
 * take a look icons from ajenti ;)
 
-
 * Disable services is_active should be computed on the fly in order to distinguish account.is_active from service.is_active when reactivation.
     * Perhaps it is time to create a ServiceModel ?
 
-
-* COpy account.main_user.username to account.name for performance
-
-* service backend execution dependency? first create user on NIS master then create directories on service server
-
 * prevent deletion of main user by the user itself
 
-
-
 * AccountAdminMixin auto adds 'account__name' on searchfields and handle account_link on fieldsets
 
 * Separate panel from server passwords?  Store passwords on panel? set_password special backend operation?
 
 * be more explicit about which backends are resources and which are service handling
 
-
 * What fields we really need on contacts? name email phone and what more?
 
-
 * Redirect junk emails and delete every 30 days?
 
 * DOC: Complitely decouples scripts execution, billing, service definition
@@ -149,24 +129,12 @@ Remember that, as always with QuerySets, any subsequent chained methods which im
     * Unify all users
 
 
-* backend admin message with link
-
 * delete main user -> delete account or prevent delete main user
 
-
-APPS app?
-
 * https://blog.flameeyes.eu/2011/01/mostly-unknown-openssh-tricks
-
 * Ansible orchestration *method* (methods.py)
-* interdependency user <-> account with the old usermodel
-
-
 * pip upgrade or install
-
-
-* disable account triggers save on cascade to execute backends save(update_field=[])
-
-
-* validate database user names
 * multiple domains creation; line separated domains
+* Move MU webapps to SaaS?
+
+* DN: Transaction atomicity and backend failure

orchestra/apps/accounts/models.py

@@ -49,6 +49,10 @@ class Account(auth.AbstractBaseUser):
     def is_staff(self):
         return self.is_superuser
     
+    @property
+    def main_systemuser(self):
+        return self.systemusers.get(is_main=True)
+    
     @classmethod
     def get_main(cls):
         return cls.objects.get(pk=settings.ACCOUNTS_MAIN_PK)
@@ -63,7 +67,8 @@ class Account(auth.AbstractBaseUser):
         created = not self.pk
         super(Account, self).save(*args, **kwargs)
         if created and hasattr(self, 'systemusers'):
-            self.systemusers.create_user(self.username, account=self, password=self.password, is_main=True)
+            self.systemusers.create(username=self.username, account=self,
+                    password=self.password, is_main=True)
     
     def disable(self):
         self.is_active = False

orchestra/apps/domains/backends.py

@@ -15,6 +15,7 @@ class Bind9MasterDomainBackend(ServiceController):
         ('domains.Record', 'domain__origin'),
         ('domains.Domain', 'origin'),
     )
+    ignore_fields = ['serial']
     
     @classmethod
     def is_main(cls, obj):
@@ -25,6 +26,7 @@ class Bind9MasterDomainBackend(ServiceController):
     def save(self, domain):
         context = self.get_context(domain)
         domain.refresh_serial()
+        print domain.render_zone()
         context['zone'] = ';; %(banner)s\n' % context
         context['zone'] += domain.render_zone()
         self.append("{ echo -e '%(zone)s' | diff -N -I'^;;' %(zone_path)s - ; } ||"

orchestra/apps/domains/models.py

@@ -1,5 +1,4 @@
 from django.db import models
-from django.utils.functional import cached_property
 from django.utils.translation import ugettext_lazy as _
 
 from orchestra.core import services

orchestra/apps/domains/serializers.py

@@ -34,6 +34,8 @@ class DomainSerializer(AccountSerializerMixin, serializers.HyperlinkedModelSeria
             try:
                 validators.validate_zone(domain.render_zone())
             except ValidationError as err:
-                self._errors = { 'all': err.message }
+                self._errors = {
+                    'all': err.message
+                }
                 return None
         return instance

orchestra/apps/domains/tests/functional_tests/tests.py

@@ -8,7 +8,7 @@ from django.core.urlresolvers import reverse
 from selenium.webdriver.support.select import Select
 
 from orchestra.apps.orchestration.models import Server, Route
-from orchestra.utils.tests import BaseLiveServerTestCase, random_ascii, snapshot_on_error
+from orchestra.utils.tests import BaseLiveServerTestCase, random_ascii, snapshot_on_error, save_response_on_error
 from orchestra.utils.system import run
 
 from ... import settings, utils, backends
@@ -129,7 +129,7 @@ class DomainTestMixin(object):
             'domain_name': domain_name,
             'server_addr': server_addr
         }
-        dig_soa = 'dig @%(server_addr)s %(domain_name)s SOA|grep "\sSOA\s"'
+        dig_soa = 'dig @%(server_addr)s %(domain_name)s SOA | grep "\sSOA\s"'
         soa = run(dig_soa % context).stdout.split()
         # testdomain.org. 3600 IN SOA ns.example.com. hostmaster.example.com. 2014021100 86400 7200 2419200 3600
         self.assertEqual('%(domain_name)s.' % context, soa[0])
@@ -140,7 +140,7 @@ class DomainTestMixin(object):
         hostmaster = utils.format_hostmaster(settings.DOMAINS_DEFAULT_HOSTMASTER)
         self.assertEqual(hostmaster, soa[5])
         
-        dig_ns = 'dig @%(server_addr)s %(domain_name)s NS|grep "\sNS\s"'
+        dig_ns = 'dig @%(server_addr)s %(domain_name)s NS  |grep "\sNS\s"'
         name_servers = run(dig_ns % context).stdout
         ns_records = ['ns1.%s.' % self.domain_name, 'ns2.%s.' % self.domain_name]
         self.assertEqual(2, len(name_servers.splitlines()))
@@ -153,7 +153,7 @@ class DomainTestMixin(object):
             self.assertEqual('NS', ns[3])
             self.assertIn(ns[4], ns_records)
         
-        dig_mx = 'dig @%(server_addr)s %(domain_name)s MX|grep "\sMX\s"'
+        dig_mx = 'dig @%(server_addr)s %(domain_name)s MX | grep "\sMX\s"'
         mx = run(dig_mx % context).stdout.split()
         # testdomain.org. 3600 IN MX 10 orchestra.lan.
         self.assertEqual('%(domain_name)s.' % context, mx[0])
@@ -163,7 +163,7 @@ class DomainTestMixin(object):
         self.assertIn(mx[4], ['30', '40'])
         self.assertIn(mx[5], ['mail3.orchestra.lan.', 'mail4.orchestra.lan.'])
         
-        dig_cname = 'dig @%(server_addr)s www.%(domain_name)s CNAME|grep "\sCNAME\s"'
+        dig_cname = 'dig @%(server_addr)s www.%(domain_name)s CNAME | grep "\sCNAME\s"'
         cname = run(dig_cname % context).stdout.split()
         # testdomain.org. 3600 IN MX 10 orchestra.lan.
         self.assertEqual('www.%(domain_name)s.' % context, cname[0])
@@ -194,13 +194,13 @@ class DomainTestMixin(object):
         self.add(self.ns1_name, self.ns1_records)
         self.add(self.ns2_name, self.ns2_records)
         self.add(self.domain_name, self.domain_records)
-        self.addCleanup(partial(self.delete, self.domain_name))
+#        self.addCleanup(partial(self.delete, self.domain_name))
         self.update(self.domain_name, self.domain_update_records)
-        self.add(self.www_name, self.www_records)
+#        self.add(self.www_name, self.www_records)
         time.sleep(0.5)
         self.validate_update(self.MASTER_SERVER_ADDR, self.domain_name)
-        time.sleep(5)
+#        time.sleep(5)
-        self.validate_update(self.SLAVE_SERVER_ADDR, self.domain_name)
+#        self.validate_update(self.SLAVE_SERVER_ADDR, self.domain_name)
     
     def test_add_add_delete_delete(self):
         self.add(self.ns1_name, self.ns1_records)
@@ -276,15 +276,18 @@ class RESTDomainMixin(DomainTestMixin):
         self.rest_login()
         self.add_route()
     
+    @save_response_on_error
     def add(self, domain_name, records):
         records = [ dict(type=type, value=value) for type,value in records ]
         self.rest.domains.create(name=domain_name, records=records)
     
+    @save_response_on_error
     def delete(self, domain_name):
         domain = Domain.objects.get(name=domain_name)
         domain = self.rest.domains.retrieve(id=domain.pk)
         domain.delete()
     
+    @save_response_on_error
     def update(self, domain_name, records):
         records = [ dict(type=type, value=value) for type,value in records ]
         domains = self.rest.domains.retrieve(name=domain_name)

orchestra/apps/lists/tests/functional_tests/tests.py

@@ -64,7 +64,7 @@ class ListMixin(object):
         backend = backends.MailmanBackend.get_name()
         Route.objects.create(backend=backend, match=True, host=server)
     
-    def atest_add(self):
+    def test_add(self):
         name = '%s_list' % random_ascii(10)
         password = '@!?%spppP001' % random_ascii(5)
         admin_email = 'root@test3.orchestra.lan'
@@ -100,8 +100,8 @@ class RESTListMixin(ListMixin):
         self.rest.lists.create(name=name, password=password, admin_email=admin_email, **extra)
     
     @save_response_on_error
-    def delete(self, username):
+    def delete(self, name):
-        list = self.rest.lists.retrieve(name=username).get()
+        list = self.rest.lists.retrieve(name=name).get()
         list.delete()
 
 

orchestra/apps/orchestration/manager.py

@@ -14,6 +14,7 @@ logger = logging.getLogger(__name__)
 
 def as_task(execute):
     def wrapper(*args, **kwargs):
+        """ failures on the backend execution doesn't fuck the request transaction atomicity """
         db.transaction.set_autocommit(False)
         try:
             log = execute(*args, **kwargs)

orchestra/apps/orchestration/middlewares.py

@@ -49,6 +49,7 @@ class OperationsMiddleware(object):
         request = getattr(cls.thread_locals, 'request', None)
         if request is None:
             return
+        good_action = action
         pending_operations = cls.get_pending_operations()
         for backend in ServiceBackend.get_backends():
             instance = None
@@ -75,7 +76,7 @@ class OperationsMiddleware(object):
                     if update_fields:
                         # "update_fileds=[]" is a convention for explicitly executing backend
                         # i.e. account.disable()
-                        if not update_fields == []:
+                        if update_fields != []:
                             execute = False
                             for field in update_fields:
                                 if field not in backend.ignore_fields:
@@ -84,12 +85,17 @@ class OperationsMiddleware(object):
                             if not execute:
                                 continue
                 instance = copy.copy(instance)
+                good = instance
                 operation = Operation.create(backend, instance, action)
                 if action != Operation.DELETE:
                     # usually we expect to be using last object state,
                     # except when we are deleting it
                     pending_operations.discard(operation)
                 pending_operations.add(operation)
+        try:
+            print kwargs['instance'], good_action
+        except:
+            pass
 
     def process_request(self, request):
         """ Store request on a thread local variable """

orchestra/apps/systemusers/tests/functional_tests/tests.py

@@ -82,9 +82,9 @@ class SystemUserMixin(object):
         # Home will be deleted on account delete, see test_delete_account
     
     def validate_ftp(self, username, password):
-        connection = ftplib.FTP(self.MASTER_SERVER)
+        ftp = ftplib.FTP(self.MASTER_SERVER)
-        connection.login(user=username, passwd=password)
+        ftp.login(user=username, passwd=password)
-        connection.close()
+        ftp.close()
         
     def validate_sftp(self, username, password):
         transport = paramiko.Transport((self.MASTER_SERVER, 22))

orchestra/apps/webapps/backends/__init__.py

@@ -1,23 +1,31 @@
 import pkgutil
-
+import textwrap
 
 class WebAppServiceMixin(object):
     model = 'webapps.WebApp'
     
     def create_webapp_dir(self, context):
-        self.append("mkdir -p '%(app_path)s'" % context)
+        self.append(textwrap.dedent("""
-        self.append("chown %(user)s.%(group)s '%(app_path)s'" % context)
+            path=""
+            for dir in $(echo %(app_path)s | tr "/" "\n"); do
+                path="${path}/${dir}"
+                [ -d $path ] || {
+                    mkdir "${path}"
+                    chown %(user)s.%(group)s "${path}"
+                }
+            done
+        """ % context))
     
     def delete_webapp_dir(self, context):
         self.append("rm -fr %(app_path)s" % context)
     
     def get_context(self, webapp):
         return {
-            'user': webapp.account.user.username,
+            'user': webapp.account.username,
-            'group': webapp.account.user.username,
+            'group': webapp.account.username,
             'app_name': webapp.name,
             'type': webapp.type,
-            'app_path': webapp.get_path(),
+            'app_path': webapp.get_path().rstrip('/'),
             'banner': self.get_banner(),
         }
 

orchestra/apps/webapps/backends/awstats.py

@@ -1,9 +0,0 @@
-from django.utils.translation import ugettext_lazy as _
-
-from orchestra.apps.orchestration import ServiceController
-
-from . import WebAppServiceMixin
-
-
-class AwstatsBackend(WebAppServiceMixin, ServiceController):
-    verbose_name = _("Awstats")

orchestra/apps/webapps/backends/phpfcgid.py

@@ -1,4 +1,5 @@
 import os
+import textwrap
 
 from django.utils.translation import ugettext_lazy as _
 
@@ -15,9 +16,12 @@ class PHPFcgidBackend(WebAppServiceMixin, ServiceController):
         context = self.get_context(webapp)
         self.create_webapp_dir(context)
         self.append("mkdir -p %(wrapper_dir)s" % context)
-        self.append(
+        self.append(textwrap.dedent("""\
-            "{ echo -e '%(wrapper_content)s' | diff -N -I'^\s*#' %(wrapper_path)s - ; } ||"
+            {
-            "  { echo -e '%(wrapper_content)s' > %(wrapper_path)s; UPDATED=1; }" % context)
+                echo -e '%(wrapper_content)s' | diff -N -I'^\s*#' %(wrapper_path)s -
+            } || {
+                echo -e '%(wrapper_content)s' > %(wrapper_path)s; UPDATED_APACHE=1
+            }""" % context))
         self.append("chmod +x %(wrapper_path)s" % context)
         self.append("chown -R %(user)s.%(group)s %(wrapper_dir)s" % context)
     
@@ -25,6 +29,10 @@ class PHPFcgidBackend(WebAppServiceMixin, ServiceController):
         context = self.get_context(webapp)
         self.delete_webapp_dir(context)
     
+    def commit(self):
+        super(PHPFcgidBackend, self).commit()
+        self.append("[[ $UPDATED_APACHE == 1 ]] && { /etc/init.d/apache reload; }")
+    
     def get_context(self, webapp):
         context = super(PHPFcgidBackend, self).get_context(webapp)
         init_vars = webapp.get_php_init_vars()
@@ -36,12 +44,12 @@ class PHPFcgidBackend(WebAppServiceMixin, ServiceController):
             context['init_vars'] = ''
         wrapper_path = settings.WEBAPPS_FCGID_PATH % context
         context.update({
-            'wrapper_content': (
+            'wrapper_content': textwrap.dedent("""\
-                "#!/bin/sh\n"
+                #!/bin/sh
-                "# %(banner)s\n"
+                # %(banner)s
-                "export PHPRC=/etc/%(type)s/cgi/\n"
+                export PHPRC=/etc/%(type)s/cgi/
-                "exec /usr/bin/%(type)s-cgi %(init_vars)s\n"
+                exec /usr/bin/%(type)s-cgi %(init_vars)s
-                ) % context,
+                """ % context),
             'wrapper_path': wrapper_path,
             'wrapper_dir': os.path.dirname(wrapper_path),
         })

orchestra/apps/webapps/backends/phpfpm.py

@@ -36,8 +36,8 @@ class PHPFPMBackend(WebAppServiceMixin, ServiceController):
         })
         context['fpm_listen'] = settings.WEBAPPS_FPM_LISTEN % context
         fpm_config = Template(
-            "[{{ user }}]\n"
             ";; {{ banner }}\n"
+            "[{{ user }}]\n"
             "user = {{ user }}\n"
             "group = {{ group }}\n\n"
             "listen = {{ fpm_listen | safe }}\n"

orchestra/apps/webapps/models.py

@@ -57,7 +57,7 @@ class WebApp(models.Model):
         return init_vars
     
     def get_fpm_port(self):
-        return settings.WEBAPPS_FPM_START_PORT + self.account.user.pk
+        return settings.WEBAPPS_FPM_START_PORT + self.account.pk
     
     def get_method(self):
         method = settings.WEBAPPS_TYPES[self.type]
@@ -66,7 +66,7 @@ class WebApp(models.Model):
     
     def get_path(self):
         context = {
-            'user': self.account.user,
+            'user': self.account.username,
             'app_name': self.name,
         }
         return settings.WEBAPPS_BASE_ROOT % context

orchestra/apps/webapps/tests/functional_tests/tests.py

@@ -0,0 +1,181 @@
+import ftplib
+import os
+import time
+import textwrap
+from StringIO import StringIO
+
+from django.conf import settings as djsettings
+from django.contrib.contenttypes.models import ContentType
+from django.core.management.base import CommandError
+from django.core.urlresolvers import reverse
+from selenium.webdriver.support.select import Select
+
+from orchestra.apps.accounts.models import Account
+from orchestra.apps.domains.models import Domain
+from orchestra.apps.orchestration.models import Server, Route
+from orchestra.apps.resources.models import Resource
+from orchestra.apps.systemusers.backends import SystemUserBackend
+from orchestra.utils.system import run, sshrun
+from orchestra.utils.tests import BaseLiveServerTestCase, random_ascii, snapshot_on_error, save_response_on_error
+
+from ... import backends, settings
+from ...models import WebApp
+
+
+class WebAppMixin(object):
+    MASTER_SERVER = os.environ.get('ORCHESTRA_MASTER_SERVER', 'localhost')
+    DEPENDENCIES = (
+        'orchestra.apps.orchestration',
+        'orchestra.apps.systemusers',
+        'orchestra.apps.webapps',
+    )
+    
+    def setUp(self):
+        super(WebAppMixin, self).setUp()
+        self.add_route()
+        djsettings.DEBUG = True
+    
+    def add_route(self):
+#        backends = [
+#            # TODO MU apps on SaaS?
+#            backends.awstats.AwstatsBackend,
+#            backends.dokuwikimu.DokuWikiMuBackend,
+#            backends.drupalmu.DrupalMuBackend,
+#            backends.phpfcgid.PHPFcgidBackend,
+#            backends.phpfpm.PHPFPMBackend,
+#            backends.static.StaticBackend,
+#            backends.wordpressmu.WordpressMuBackend,
+#        ]
+        server = Server.objects.create(name=self.MASTER_SERVER)
+        for backend in [SystemUserBackend, self.backend]:
+            backend = backend.get_name()
+            Route.objects.create(backend=backend, match=True, host=server)
+    
+    def test_add(self):
+        name = '%s_%s_webapp' % (random_ascii(10), self.type_value)
+        self.add_webapp(name)
+        self.validate_add_webapp(name)
+#        self.addCleanup(self.delete, username)
+
+
+class StaticWebAppMixin(object):
+    backend = backends.static.StaticBackend
+    type_value = 'static'
+    token = random_ascii(100)
+    page = (
+        'index.html',
+        '<html>Hello World! %s </html>\n' % token,
+        '<html>Hello World! %s </html>\n' % token,
+    )
+    
+    def validate_add_webapp(self, name):
+        try:
+            ftp = ftplib.FTP(self.MASTER_SERVER)
+            ftp.login(user=self.account.username, passwd=self.account_password)
+            ftp.cwd('webapps/%s' % name)
+            index = StringIO()
+            index.write(self.page[1])
+            index.seek(0)
+            ftp.storbinary('STOR %s' % self.page[0], index)
+            index.close()
+        finally:
+            ftp.close()
+
+
+class PHPFcidWebAppMixin(StaticWebAppMixin):
+    backend = backends.phpfcgid.PHPFcgidBackend
+    type_value = 'php5'
+    token = random_ascii(100)
+    page = (
+        'index.php',
+        '<?php print("Hello World! %s");\n?>\n' % token,
+        'Hello World! %s' % token,
+    )
+
+
+class PHPFPMWebAppMixin(StaticWebAppMixin):
+    backend = backends.phpfpm.PHPFPMBackend
+    type_value = 'php5.5'
+    token = random_ascii(100)
+    page = (
+        'index.php',
+        '<?php print("Hello World! %s");\n?>\n' % token,
+        'Hello World! %s' % token,
+    )
+
+
+class RESTWebAppMixin(object):
+    def setUp(self):
+        super(RESTWebAppMixin, self).setUp()
+        self.rest_login()
+        # create main user
+        self.save_systemuser()
+    
+    @save_response_on_error
+    def save_systemuser(self):
+        self.rest.systemusers.retrieve().get().save()
+    
+    @save_response_on_error
+    def add_webapp(self, name, options=[]):
+        self.rest.webapps.create(name=name, type=self.type_value)
+    
+    @save_response_on_error
+    def delete_webapp(self, name):
+        list = self.rest.lists.retrieve(name=name).get()
+        list.delete()
+
+
+class AdminWebAppMixin(WebAppMixin):
+    def setUp(self):
+        super(AdminWebAppMixin, self).setUp()
+        self.admin_login()
+        # create main user
+        self.save_systemuser()
+    # TODO save_account()
+    
+    @snapshot_on_error
+    def add(self, name, password, admin_email):
+        url = self.live_server_url + reverse('admin:mails_List_add')
+        self.selenium.get(url)
+        
+        account_input = self.selenium.find_element_by_id('id_account')
+        account_select = Select(account_input)
+        account_select.select_by_value(str(self.account.pk))
+        
+        name_field = self.selenium.find_element_by_id('id_name')
+        name_field.send_keys(username)
+        
+        password_field = self.selenium.find_element_by_id('id_password1')
+        password_field.send_keys(password)
+        password_field = self.selenium.find_element_by_id('id_password2')
+        password_field.send_keys(password)
+        
+        if quota is not None:
+            quota_id = 'id_resources-resourcedata-content_type-object_id-0-allocated'
+            quota_field = self.selenium.find_element_by_id(quota_id)
+            quota_field.clear()
+            quota_field.send_keys(quota)
+        
+        if filtering is not None:
+            filtering_input = self.selenium.find_element_by_id('id_filtering')
+            filtering_select = Select(filtering_input)
+            filtering_select.select_by_value("CUSTOM")
+            filtering_inline = self.selenium.find_element_by_id('fieldsetcollapser0')
+            filtering_inline.click()
+            time.sleep(0.5)
+            filtering_field = self.selenium.find_element_by_id('id_custom_filtering')
+            filtering_field.send_keys(filtering)
+        
+        name_field.submit()
+        self.assertNotEqual(url, self.selenium.current_url)
+
+
+class RESTWebAppTest(PHPFcidWebAppMixin, RESTWebAppMixin, WebAppMixin, BaseLiveServerTestCase):
+    pass
+
+
+#class AdminWebAppTest(AdminWebAppMixin, BaseLiveServerTestCase):
+#    pass
+
+
+

orchestra/apps/websites/backends/apache.py

@@ -30,15 +30,13 @@ class Apache2Backend(ServiceController):
         apache_conf = Template(textwrap.dedent("""\
             # {{ banner }}
             <VirtualHost *:{{ site.port }}>
-                ServerName {{ site.domains.all|first }}
+                ServerName {{ site.domains.all|first }}\
             {% if site.domains.all|slice:"1:" %}
-                ServerAlias {{ site.domains.all|slice:"1:"|join:' ' }}
+                ServerAlias {{ site.domains.all|slice:"1:"|join:' ' }}{% endif %}
-            {% endif %}
                 CustomLog {{ logs }} common
-                SuexecUserGroup {{ user }} {{ group }}
+                SuexecUserGroup {{ user }} {{ group }}\
-            {% for line in extra_conf.splitlines %}"
+            {% for line in extra_conf.splitlines %}
-                {{ line | safe }}
+                {{ line | safe }}{% endfor %}
-            {% endfor %}
             </VirtualHost>"""
         ))
         apache_conf = apache_conf.render(Context(context))
@@ -83,13 +81,13 @@ class Apache2Backend(ServiceController):
         context = self.get_content_context(content)
         context['fcgid_path'] = fcgid_path % context
         fcgid = self.get_alias_directives(content)
-        fcgid += (
+        fcgid += textwrap.dedent("""\
-            "ProxyPass %(location)s !\n"
+            ProxyPass %(location)s !
-            "<Directory %(app_path)s>\n"
+            <Directory %(app_path)s>
-            "    Options +ExecCGI\n"
+                Options +ExecCGI
-            "    AddHandler fcgid-script .php\n"
+                AddHandler fcgid-script .php
-            "    FcgidWrapper %(fcgid_path)s\n"
+                FcgidWrapper %(fcgid_path)s
-        ) % context
+            """ % context)
         for option in content.webapp.options.filter(name__startswith='Fcgid'):
             fcgid += "    %s %s\n" % (option.name, option.value)
         fcgid += "</Directory>\n"
@@ -100,11 +98,11 @@ class Apache2Backend(ServiceController):
         custom_cert = site.options.filter(name='ssl')
         if custom_cert:
             cert = tuple(custom_cert[0].value.split())
-        directives = (
+        directives = textwrap.dedent("""\
-            "SSLEngine on\n"
+            SSLEngine on
-            "SSLCertificateFile %s\n"
+            SSLCertificateFile %s
-            "SSLCertificateKeyFile %s\n"
+            SSLCertificateKeyFile %s""" % cert
-        ) % cert
+        )
         return directives
     
     def get_security(self, site):
@@ -129,17 +127,17 @@ class Apache2Backend(ServiceController):
             path, name, passwd = re.match(regex, protection.value).groups()
             path = os.path.join(context['root'], path)
             passwd = os.path.join(self.USER_HOME % context, passwd)
-            protections += ("\n"
+            protections += textwrap.dedent("""
-                "<Directory %s>\n"
+                <Directory %s>
-                "    AllowOverride All\n"
+                    AllowOverride All
-#                "    AuthPAM_Enabled off\n"
+                    #AuthPAM_Enabled off
-                "    AuthType Basic\n"
+                    AuthType Basic
-                "    AuthName %s\n"
+                    AuthName %s
-                "    AuthUserFile %s\n"
+                    AuthUserFile %s
-                "    <Limit GET POST>\n"
+                    <Limit GET POST>
-                "        require valid-user\n"
+                        require valid-user
-                "    </Limit>\n"
+                    </Limit>
-                "</Directory>\n" % (path, name, passwd)
+                </Directory>""" % (path, name, passwd)
             )
         return protections
     
@@ -161,8 +159,8 @@ class Apache2Backend(ServiceController):
             'site': site,
             'site_name': site.name,
             'site_unique_name': site.unique_name,
-            'user': site.account.user.username,
+            'user': site.account.username,
-            'group': site.account.user.username,
+            'group': site.account.username,
             'sites_enabled': sites_enabled,
             'sites_available': "%s.conf" % os.path.join(sites_available, site.unique_name),
             'logs': os.path.join(settings.WEBSITES_BASE_APACHE_LOGS, site.unique_name),
@@ -190,7 +188,7 @@ class Apache2Traffic(ServiceMonitor):
     def prepare(self):
         current_date = timezone.localtime(self.current_date)
         current_date = current_date.strftime("%Y%m%d%H%M%S")
-        self.append(textwrap.dedent("""
+        self.append(textwrap.dedent("""\
             function monitor () {
                 OBJECT_ID=$1
                 INI_DATE=$2

orchestra/apps/websites/tests/functional_tests/tests.py

@@ -0,0 +1,91 @@
+import os
+import socket
+import time
+import textwrap
+
+from django.conf import settings as djsettings
+from django.contrib.contenttypes.models import ContentType
+from django.core.management.base import CommandError
+from django.core.urlresolvers import reverse
+import requests
+from selenium.webdriver.support.select import Select
+
+from orchestra.apps.accounts.models import Account
+from orchestra.apps.domains.models import Domain, Record
+from orchestra.apps.domains.backends import Bind9MasterDomainBackend
+from orchestra.apps.orchestration.models import Server, Route
+from orchestra.apps.resources.models import Resource
+from orchestra.apps.webapps.tests.functional_tests.tests import StaticWebAppMixin, RESTWebAppMixin, WebAppMixin, PHPFcidWebAppMixin, PHPFPMWebAppMixin
+from orchestra.utils.system import run, sshrun
+from orchestra.utils.tests import BaseLiveServerTestCase, random_ascii, snapshot_on_error, save_response_on_error
+
+from ... import backends, settings
+from ...models import Website
+
+
+class WebsiteMixin(WebAppMixin):
+    MASTER_SERVER = os.environ.get('ORCHESTRA_MASTER_SERVER', 'localhost')
+    MASTER_SERVER_ADDR = socket.gethostbyname(MASTER_SERVER)
+    DEPENDENCIES = (
+        'orchestra.apps.orchestration',
+        'orchestra.apps.domains',
+        'orchestra.apps.websites',
+        'orchestra.apps.webapps',
+        'orchestra.apps.systemusers',
+    )
+    
+    def add_route(self):
+        super(WebsiteMixin, self).add_route()
+        server = Server.objects.get()
+        backend = backends.apache.Apache2Backend.get_name()
+        Route.objects.create(backend=backend, match=True, host=server)
+        backend = Bind9MasterDomainBackend.get_name()
+        Route.objects.create(backend=backend, match=True, host=server)
+    
+    def validate_add_website(self, name, domain):
+        url = 'http://%s/%s' % (domain.name, self.page[0])
+        self.assertEqual(self.page[2], requests.get(url).content)
+    
+    def test_add(self):
+        # TODO domains with "_" bad name!
+        domain_name = '%sdomain.lan' % random_ascii(10)
+        domain = Domain.objects.create(name=domain_name, account=self.account)
+        domain.records.create(type=Record.A, value=self.MASTER_SERVER_ADDR)
+        self.save_domain(domain)
+        webapp = '%s_%s_webapp' % (random_ascii(10), self.type_value)
+        self.add_webapp(webapp)
+        self.validate_add_webapp(webapp)
+        website = '%s_website' % random_ascii(10)
+        self.add_website(website, domain, webapp)
+        self.validate_add_website(website, domain)
+
+
+class RESTWebsiteMixin(RESTWebAppMixin):
+    @save_response_on_error
+    def save_domain(self, domain):
+        self.rest.domains.retrieve().get().save()
+    
+    def add_website(self, name, domain, webapp):
+        domain = self.rest.domains.retrieve().get()
+        webapp = self.rest.webapps.retrieve().get()
+        self.rest.websites.create(name=name, domains=[domain.url], contents=[{'webapp': webapp.url}])
+
+
+
+#class RESTWebsiteTest(RESTWebsiteMixin, StaticWebAppMixin, WebsiteMixin, BaseLiveServerTestCase):
+#    pass
+
+
+PHPFPMWebAppMixin
+#class RESTWebsiteTest(RESTWebsiteMixin, PHPFcidWebAppMixin, WebsiteMixin, BaseLiveServerTestCase):
+#    pass
+
+
+class RESTWebsiteTest(RESTWebsiteMixin, PHPFPMWebAppMixin, WebsiteMixin, BaseLiveServerTestCase):
+    pass
+
+#class AdminWebsiteTest(AdminWebsiteMixin, BaseLiveServerTestCase):
+#    pass
+
+
+

orchestra/core/validators.py

@@ -41,7 +41,7 @@ def validate_name(value):
     """
     A single non-empty line of free-form text with no whitespace.
     """
-    validators.RegexValidator('^\w+$',
+    validators.RegexValidator('^[\.\w]+$',
         _("Enter a valid name (text without whitspaces)."), 'invalid')(value)
 
 

scripts/services/apache_full_stack.md

@@ -28,12 +28,14 @@ The goal of this setup is having a high-performance state-of-the-art deployment
     apt-get install apache2-mpm-event php5-fpm libapache2-mod-fcgid apache2-suexec-custom php5-cgi
     ```
 
+# TODO libapache2-mod-auth-pam is no longer part of the debian distribution,
+#       replace with libapache2-mod-authnz-external pwauth
 
 2. Enable some convinient Apache modules
     ```bash
     a2enmod suexec
     a2enmod ssl
-    a2enmod auth_pam
+    #a2enmod auth_pam
     a2enmod proxy_fcgi
     a2emmod userdir
     ```
@@ -55,15 +57,18 @@ The goal of this setup is having a high-performance state-of-the-art deployment
     ```
 
 
+
+
+
 5. Restart Apache
     ```bash
     service apache2 restart
     ```
 
 
-* TODO 
+
-    libapache2-mod-auth-pam
+
-    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=710770
+
 
 
 * ExecCGI
@@ -73,6 +78,11 @@ The goal of this setup is having a high-performance state-of-the-art deployment
     </Directory>
     ```
 
+* Permissions
+<Directory /home/*/webapps>
+        Require all granted
+</Directory>
+
 
 TODO CHRoot
     https://andrewbevitt.com/tutorials/apache-varnish-chrooted-php-fpm-wordpress-virtual-host/

scripts/services/vsftpd.md

@@ -10,10 +10,12 @@ VsFTPd with System Users
 
 2. Make some configurations
     ```bash
-    sed -i "s/anonymous_enable=YES/anonymous_enable=NO/" /etc/vsftpd.conf
+    sed -i "s/^anonymous_enable=YES/anonymous_enable=NO/" /etc/vsftpd.conf
-    sed -i "s/#local_enable=YES/local_enable=YES/" /etc/vsftpd.conf
+    sed -i "s/^#local_enable=YES/local_enable=YES/" /etc/vsftpd.conf
-    sed -i "s/#write_enable=YES/write_enable=YES/" /etc/vsftpd.conf
+    sed -i "s/^#write_enable=YES/write_enable=YES/" /etc/vsftpd.conf
-    # sed -i "s/#chroot_local_user=YES/chroot_local_user=YES/" /etc/vsftpd.conf
+    # sed -i "s/^#chroot_local_user=YES/chroot_local_user=YES/" /etc/vsftpd.conf
+    
+    sed -i "s/^#local_umask=022/local_umask=022/" /etc/vsftpd.conf
     
     echo '/dev/null' >> /etc/shells
     ```