3 May 2019 / TOOLS, TYPO3, DOCKER

A modern way to organize and build your TYPO3 Site with Composer and Webpack Encore

Alex Liivet from Chester, United Kingdom Creative Commons CC0 1.0 Universal Public Domain Dedication, via Wikimedia Commons

The way we are using TYPO3 has constantly been changed in the last 10 years. I will show you, how we use state-of-the-art technologies to optimize and automatize the way of building and deploying our TYPO3 sites.

History

There were times no source code repository existed and everything was done live on the server. Then we had monolithic repositories in Subversion (do you member?) where each and everything belonging to the site was stored.

When git emerged some sites used a first kind of dependency management with the heavy usage of git submodules, that worked quite well, but still some manual work to get the job done. The next step was using a build server (Jenkins) to manage and deploy all the separate git repositories to the desired location.

With the rise of Composer and Docker we found a great way to build our site using Jenkins or Gitlab CI, but we still had two separate repositories for one site. On the one hand we had to maintain the root repository with the root-composer.json and another repository for the sitepackage (a TYPO3 extension).

Problems that arose out of this constellation were a shaky way to compile and build assets (that ended in committing compiled assets, such as JavaScipt and CSS files to the git repository) and always running composer update when something changed in the sitepackage.

Today

As we like working with Symfony and are used to the structures Symfony provides, we changed the way we were thinking of a TYPO3 website and handled it as a modern web application. Our root package structure looks like that (I omitted some irrelevant files):

├── assets
├── config
├── packages
├── public
├── .env
├── .gitlab-ci.yml
├── composer.json
├── composer.lock
├── Dockerfile
├── package.json
├── postcss.config.js
├── README.md
├── webpack.config.js
└── yarn.lock

That means, all code and assets that belong to the site and are managed by the site (assets, TypoScript configuration, Hooks, ViewHelpers, …) have to be located in a single package (repository). The package contains a root-composer.json where all dependencies are stored.

Sitepackage a.k.a. Template Extension

The sitepackage unfortunatly has to be a TYPO3 Extension, because some things, such as referencing assets, templates and so on (currently) do not work when they are stored outside of the document root.

So, the sitepackage is still an Extension, but stored in /packages/tmpl_foo. It contains a basic composer.json file:

{
    "name": "subugoe/tmpl-foo",
    "description": "Foo Website template",
    "type": "typo3-cms-extension",
    "license": "GPL-2.0-or-later",
    "authors": [
        {
            "name": "Ingo Pfennigstorf",
            "email": "pfennigstorf@sub.uni-goettingen.de"
        }
    ],
    "autoload": {
        "psr-4": {
             "Subugoe\\TmplFoo\\": "Classes/"
        }
    },
    "require": {
        "typo3/cms-core": "^9.5"
    }
}

To get that working in TYPO3 we use basic Composer methods, that allows us to use repositories not only referenced in packagist, but also local paths.

{
  "name": "subugoe/foo-site",
  "description": "Foo Website package",
  "license": "GPL-2.0-or-later",
  "type": "project",
  "repositories": [
    {
      "type": "path",
      "url": "./packages/tmpl_foo/"
    }
  ],
  "require": {
    "georgringer/news": "^7.0.6",
    "helhum/dotenv-connector": "^2.0",
    "helhum/typo3-console": "^5.5",
    "subugoe/tmpl-foo": "dev-develop",
    "typo3/cms-beuser": "^9.5",
    "typo3/cms-fluid-styled-content": "^9.5",
    "typo3/cms-form": "^9.5",
    "typo3/cms-recycler": "^9.5",
    "typo3/cms-redirects": "^9.5",
    "typo3/cms-reports": "^9.5",
    "typo3/cms-rte-ckeditor": "^9.5",
    "typo3/cms-seo": "^9.5",
    "typo3/cms-setup": "^9.5",
    "typo3/cms-t3editor": "^9.5",
    "typo3/cms-tstemplate": "^9.5",
    "typo3/cms-viewpage": "^9.5",
    "typo3/cms-workspaces": "^9.5",
    "typo3/minimal": "^9.5"
  },
  "scripts": {
    "typo3-deployment-scripts": [
      "typo3cms cache:flush"
    ],
    "typo3-cms-scripts": [
      "typo3cms install:fixfolderstructure",
      "typo3cms install:generatepackagestates"
    ],
    "post-install-cmd": [
      "@typo3-cms-scripts"
    ],
    "post-update-cmd": [
      "@typo3-cms-scripts"
    ]
  },
  "config": {
    "sort-packages": true,
    "platform": {
      "php": "7.2"
    }
  },
  "require-dev": {
    "friendsofphp/php-cs-fixer": "^2.11"
  }
}

What happens if you run composer install here? All dependencies are installed and the sitepackage from /packages/tmpl_foo will be symlinked to its final destination /public/typo3conf/ext/tmpl_foo, so it can be used in the same way as a normal extension, such as news.

If any changes in the site package need to be performed, that can be done in /packages/tmpl_foo, and as it is symlinked, the changes will be visible instantly.

Another big issue we had, was compiling and adding assets from SCSS and transpile and bundle modern JavaScript in a way that browsers understand that.

Assets with Webpack Encore

My aim here is, to only commit the original SCSS and JavaScript files and the rest, such as compiling and transpiling will be done in the CI-step. In the past I tried to avoid Webpack and use Gulp, because honestly, I didn’t understand webpack at all. But then came Webpack Encore to the rescue.

Encore is basically designed to manage assets in Symfony environments, but as we treat our tiny TYPO3 Website as an application that also core-wise uses loads of Symfony components I also gave it a shot, and it’s totally worth it. Encore is a wrapper around Webpack that has convenience-methods for many tasks, e.g compiling SCSS into CSS and ES201x JavaScript into a bowser readable way.

I recently used yarn for handling the JavaScript dependencies and for running the building tasks. So, for adding encore to our package.json where all other dependencies are declared (Bootstrap, Vue, jQuery, …) we run yarn add @symfony/webpack-encore --dev and create a webpack.config.json file on the root level. I will not go too deep here, as there is a good documentation about that.

All assets are stored in /assets/{js/images/scss} and compiled when running yarn run encore production. The compiled assets are moved into the sitepackage, and to achieve that, the webpack.config.json looks so:

var Encore = require('@symfony/webpack-encore');

Encore
    // directory where compiled assets will be stored
    .setOutputPath('./public/typo3conf/ext/tmpl_foo/Resources/Public/build/')
    // public path used by the web server to access the output path
    .setPublicPath('/typo3conf/ext/tmpl_foo/Resources/Public/build/')
    // only needed for CDN's or sub-directory deploy
    //.setManifestKeyPrefix('build/')

    /*
     * ENTRY CONFIG
     *
     * Add 1 entry for each "page" of your app
     * (including one that's included on every page - e.g. "app")
     *
     * Each entry will result in one JavaScript file (e.g. app.js)
     * and one CSS file (e.g. app.css) if you JavaScript imports CSS.
     */
    .addEntry('app', './assets/js/app.js')

    // will require an extra script tag for runtime.js
    // but, you probably want this, unless you're building a single-page app
    .enableSingleRuntimeChunk()

    .cleanupOutputBeforeBuild()
    .enableSourceMaps(!Encore.isProduction())
    // enables hashed filenames (e.g. app.abc123.css)
    .enableVersioning(false)

    // uncomment if you use Sass/SCSS files
    .enableSassLoader()

    // uncomment if you're having problems with a jQuery plugin
    .autoProvidejQuery()
    .enablePostCssLoader()
;

module.exports = Encore.getWebpackConfig();

The generated files are included via TypoScript in the most easy way in the sitepacke TypoScript:

page {
    includeCSS {
        app = EXT:tmpl_foo/Resources/Public/build/app.css
        app.media = all
    }

    includeJSLibs {
        runtime = EXT:tmpl_foo/Resources/Public/build/runtime.js
        app = EXT:tmpl_foo/Resources/Public/build/app.js
    }
}

That’s basically all we need. Encore is quite good at handling erros and often provides good solutions to problems in the error messages that appear on the console.

Building with Docker and Gitlab CI

As final step the application is built on Gitlab CI that creates a self-containing Docker-Image. I wrote about that several times in this blog, so nothing new here.

The Dockerfile needs to have information about the frontend building process, that is quite new in the way we deal with our assets, so the Dockerfile looks like that:

FROM php:7.3-apache

ENV APACHE_DOCUMENT_ROOT /var/www/html/public/
ENV COMPOSER_ALLOW_SUPERUSER 1

WORKDIR /var/www/html/
COPY . /var/www/html/

RUN apt-get update -yqq && \
    apt-get install -yqq \
        libcurl4-gnutls-dev \
        libfreetype6-dev \
        libjpeg62-turbo-dev \
        libicu-dev \
        libpng-dev \
        libbz2-dev \
        libzip-dev \
        graphicsmagick \
        apt-transport-https \
        git \
        libxml2-dev \
        wget \
        gnupg \
        unzip \
        zlib1g-dev && \
    # Install PHP extensions
    docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ && \
    docker-php-ext-install -j$(nproc) \
        intl \
        gd \
        zip \
        bz2 \
        opcache \
        mysqli && \
    printf '[PHP]\ndate.timezone = "Europe/Berlin"\n' > /usr/local/etc/php/conf.d/tzone.ini && \
    pecl install apcu && \
    docker-php-ext-enable apcu && \
    # Install and run Composer
    curl -sS https://getcomposer.org/installer | php && \
    php composer.phar install \
      --prefer-dist \
      --no-progress \
      --no-suggest \
      --optimize-autoloader \
      --classmap-authoritative  \
      --no-interaction && \
    php composer.phar clear-cache && \
    # Node and yarn
    curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
    echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list && \
    curl -sL https://deb.nodesource.com/setup_11.x | bash - && \
    apt-get update -yqq && \
    apt-get install -yqq \
        nodejs \
        yarn && \
    # Build Js / CSS
    yarn install && \
    yarn run encore production && \
    # Clean up
    docker-php-source delete && \
    apt-get remove -yqq \
        yarn \
        nodejs && \
    rm -rf /var/lib/apt/lists/* && \
    rm -rf /var/www/html/node_modules && \
    chown -R www-data:www-data /var/www/html && \
    mkdir -p /var/www/html/public/typo3temp/assets/_processed_

Conclusion

I hope, I didn’t forget anything crucial, but if so, please drop me a line. This way of working currently works really well for us, but I’m sure there will always be things to optimize and new and better technologies to adapt.