Video Encoding with Large Files

This week’s issue relates to encoding large video files. I had previously integrated open source software to encode video from a variety of formats to H.264 (using an mp4 container). The software worked fine with test data from small video files. Recently however, we have been testing with more realistic data from longer videos.

Soon into this stage of testing, we found that the encoding software which was running on our test server was failing.

We use the open source software ‘melt’ on Linux to do the video encoding. ‘Melt’ usually runs as a two-pass process.

So I went to look at our logs and determined that the ‘melt’ first pass in particular was failing. Next, I checked from running ‘top’ in another terminal that melt was using more and more memory and was terminating at 92% of total memory.

As I suspected that the Linux Out-Of-Memory killer had terminated the process, I went to check the kernel logs, as follows.

$ tail /var/log/kern.log
kernel: [32941851.928772] Out of memory: Kill process 3318 (melt) score 927 or sacrifice child
kernel: [32941851.928788] Killed process 3318 (melt) total-vm:4400676kB, anon-rss:3551936kB, file-rss:140kB

This confirms that the OOM-killer terminated the melt process as memory needs to be reclaimed to ensure that Linux could continue to run.

Now that I knew the source of the problem, first I tweaked the melt parameters to reduce its workload. However, the same problem occurred again with only a slight improvement in how far the encoding went.

Following one from this, I checked our Linux box and saw that no swap space had been allocated. From here, I could see that the solution was to configure swap memory in Linux to allow the melt process to complete.

So, we allocated 2GB of swap memory in addition to the existing 4GB of RAM and ran the encoding test again. This time, everything ran smoothly, with the swap memory being used to supplement RAM.

Advertisements
Tagged with: , ,
Posted in Video Encoding

Using a virtual machine as a development environment

For my last project, I decided to use Linux as my development environment.  My laptop runs Windows and I wanted to try using Linux without setting up a dual-boot system. I tried VMware Player and installed Linux Mint with the Mate interface. I have Linux Mint with Cinnamon, so this would be a way to try a different variation.

I found the virtual machine to have good performance apart from the User Interface, which showed some lag for scrolling, especially when using MySQL Workbench, which depends on a graphical representation for the schema.

Overall, it was good to have a complete development environment which is separate from that of my main work. I configured the Apache environment to work with my repository. All of these changes are encapsulated in one folder on the host machine, which I zipped into an archive file. This makes it easier to make configuration changes as you can always roll back.

Recently, I looked at Oracle Virtual Box. This is open-source software. It looks to be quite a good program, and I plan on using it going forward for a personal project. I may move some of the work of my main project from Windows onto a Linux virtual machine, time permitting.

Tagged with: , ,
Posted in Virtual Machines

Including Autoloader and Laravel

This week, the challenge was to integrate the front and back ends. The front end code was deployed to our test server. There was a predefined folder structure for the back-end code that I had to copy my PHP code. I kept the Laravel folder structure and just copied across my code into the new structure.

Once we tested the app, the first problem was that the Laravel auto-loader “include” was not working. This was traced to the difference in the directory structure on the test server and the one that Laravel expected.

My initial thought was to change my local folder structure to match the test server. Then, I saw that the production folder structure was going to differ slightly to the test environment. So, it made sense to configure the autoloader “include” statements depending on server environment – local, staging or production.

Laravel has support for configuring different environments, and provides an app->environment() method to give the current environment.

 $environment = App::environment();

So, you can test your environment as follows.

 if (App::environment('staging')){
        ....
 }

However, I could not use the Laravel framework in this case, because the auto-loader had to be included in index.php, before the app variable was set up. In fact, the location of start.php (where $app is set up) would be different on each server.

The solution was to test the PHP global, $_SERVER.

 if( $_SERVER['SERVER_NAME'] == 'localhost') {
     // If the environment is local...
     require __DIR__.'/../bootstrap/autoload.php';
     $app = require_once __DIR__.'/../bootstrap/start.php';
 }
 elseif( $_SERVER['SERVER_NAME'] == 'staging.zzzzz.com') {
     // The environment is staging...
     require __DIR__.'/../staging_server/bootstrap/autoload.php';
     $app = require_once __DIR__.'/../staging_server /bootstrap/start.php';
 }
 elseif ( $_SERVER['SERVER_NAME'] == 'www.zzzzz.com') {
     // The environment is production...
     require __DIR__.'/../production_server/bootstrap/autoload.php';
     $app = require_once __DIR__.'/../production_server/bootstrap/start.php';
 }

 /*
 |--------------------------------------------------------------------------
 | Run The Application
 |--------------------------------------------------------------------------
 */
 $app->run();

So, in this case, I could not use Laravel’s configuration framework, as Laravel had not yet started. It was time to use plain old PHP.

Tagged with: ,
Posted in Web Development

Improving the development cycle for back-end development

My first web project was based on the LAMP stack. We used NetBeans on Windows as the code editor for writing PHP. Once the code was written, it was transferred to a remote Linux server to be served by Apache or to be run as PHP unit tests. So, the workflow consisted on coding on Windows, and then using a file transfer program such as FileZilla or WinSCP to move the PHP script to the Linux server, where it could be tested. If the original code needed to be changed after testing, we would follow the same cycle of updating the code on Windows and then moving it to the remote Linux server for testing.

We used Subversion on Windows for source code control. So, code was checked on from the repository into a Windows directory for editing.

Recently, I started a new back-end project for a different client. As this was a new project, I was able to start over with a fresh approach. It looked to me like the previous approach of developing on Windows and testing on Linux was less than optimal.

When I started the new project, I set up a LAMP stack on my computer. This time I decided to develop on NetBeans on Linux. This allowed me to test on my local Linux environment, when was running Apache 2, MySQL and PHP.

At first, I followed a similar approach to the first project. For the new project, we use Git as the source code control system. So I cloned a copy of the back-end repository to its own folder in my home directory.

Initially, I left the Apache Document Root at its default of “/var/www” and copied my PHP code there for testing. After testing I would have to move my updated PHP scripts back to their folder in my home directory so I could check them in to the Git repository.

It soon became obvious that this was a time consuming and error prone process as I had to remember which files were updated during the edit/test cycle. As we use the Laravel 4 PHP framework and follow its directory structure, it created a mental burden to remember what files had been updated, so they could be moved back to the local repository for staging and committing.

The solution was to update the Apache Document Root to point to my local repository. This would mean that there would be no need to move files back and forth, and a simple scan in Git Gui would indicate what files had changed and would need to be committed to the remote repository.

So, I made a copy of my /etc/apache2/sites-available/000-default.conf which defines the default website for Apache on Linux Mint.  In the new file (e.g. mysite.conf),  I updated the DocumentRoot directive to point to my local repository,disabled the default site, enabled my virtual site and restarted Apache.

The script to enable my new virtual host is below.

sudo a2dissite 000-default && sudo a2ensite mysite

After I restarted Apache, I attempted to load my Laravel website loaded. However, the Laravel public folder was not visible. This was crucial as my PHP code depended on Laravel. Further investigation showed that the public folder had it own .htaccess file. When I disabled this file (by re-naming it), the public folder became visible again in the browser.

This solved my immediate problem, but I still wanted my Laravel routing to work. As the problem was with .htaccess,  this implied that mod_rewite need to be enabled. Therefore the solution lay in the Apache  AllowOveride directive. This had to be changed from the default of None to FileInfo for Laravel. I added the following code to mysite.conf, and restarted Apache and all worked as intended.

<VirtualHost *:80>
…
…
          DocumentRoot  "/home/myuser/myproject/my_backend_repository"
        <Directory "/home/myuser/myproject/my_backend_repository">
          Options Indexes FollowSymLinks MultiViews
            # changed from None to FileInfo
            AllowOverride FileInfo
            Order allow,deny
            allow from all
         </Directory>
…
…
< /VirtualHost >

While there is some configuration involved, there are important advantages to using this approach over my original approach of using local editing and remote testing mentioned at the start of this blog entry.

First of all the check-in cycle is much simpler as the code is never moved out off the local repository for testing. It is easy to know what files have changed, so the correct updates are always checked in.

Secondly,  as testing is done through the local Apache server, there is no need to use file transfer to move the code from the local machine to the server. This improves workflow.

Next, development can proceed even without an Internet connection to the remote server. This can be handy is there are any networking problems.

Finally, it is also nice that the response time from the local server is always immediate.

Tagged with: , , , ,
Posted in Web Development