How to add Twitter Bootstrap to a Symfony 2 application

Note: the following assumes you are not using gulp, grunt or another workflow tool. If you are, and want to include downloading and compiling Bootstrap and JQuery into your workflow, see their websites for more information on how to do this.

  1. Download the latest version of Bootstrap from http://getbootstrap.com/.
  2. If you extract the zip file, you will notice 3 folders:
    |-css
    |-fonts
    |-js
    
  3. Copy these into your web directory so that your web directory looks like the following:

    |-web
        |-css
        |-fonts
        |-js
    
  4. Download jquery minified version from https://jquery.com/
  5. Save this file into your web/js directory
  6. In you app/Resources/views/base.html.twig file, add the following:
    <!-- in head tag -->
    <link rel="stylesheet" href="{{ asset('css/bootstrap.css') }}" />
    <link rel="stylesheet" href="{{ asset('css/bootstrap-theme.css') }}" />
    
    <!-- at end of body tag -->
    <script src="{{ asset('js/jquery.min.js') }}"></script>
    <script src="{{ asset('js/bootstrap.min.js') }}"></script>
    

    If you are using the default base.html.twig file, it will look like this:

    <!DOCTYPE html>
    <html>
        <head>
            <meta charset="UTF-8" />
            <title>{% block title %}Welcome!{% endblock %}</title>
            {% block stylesheets %}{% endblock %}
            <link rel="icon" type="image/x-icon" href="{{ asset('favicon.ico') }}" />
            <link rel="stylesheet" href="{{ asset('css/bootstrap.css') }}" />
            <link rel="stylesheet" href="{{ asset('css/bootstrap-theme.css') }}" />
        </head>
        <body>
            <h1>app/Resources/view/base.html.twig</h1>
             {% block nav %}{% endblock %}
            {% block body %}{% endblock %}
            <script src="{{ asset('js/jquery.min.js') }}"></script>
            <script src="{{ asset('js/bootstrap.min.js') }}"></script>
            {% block javascripts %}{% endblock %}
        </body>
    </html>
    

Bonus:
Symfony includes a form theme for Bootstrap out of the box. To enable Bootstrap in your forms without having to create custom templates, simply add the following to your app/config/config.yml file:

twig:
    form:
        resources: ['bootstrap_3_layout.html.twig']
        # resources: ['bootstrap_3_horizontal_layout.html.twig']

Source: http://symfony.com/blog/new-in-symfony-2-6-bootstrap-form-theme

Determining which tab is using resources

If you are like me, you usually have many tabs open at once and every so often your browser will slow to a crawl. Identifying the culprit is usually based on a lucky guess, but browsers have some in-built tools to help you.

In Firefox, type about:performance into the address bar.

In Chrome, go to the menu to the right off the address bar (3 vertical dots), select More Tools and then Task Manager.

Both of these will show you which tabs are using the most resources and allow you to close the tab from within them.

Customizing Form layout in Symfony

While working on a recent Symfony project I needed to modify the default Symfony form layout. In my case I needed to add a custom CSS class to the form.

Note: the tips below assume you are using the Twig layout engine.

1) Create a file to contain your overrides called app/resources/form/fields.html.twig.

2) Open up the base file

3) Copy the required blocks from the base file to your new fields.html.twig file an modify as needed.

4) In the forms that you want to use your new layout, add the following:
{% form_theme delete_form ‘form/fields.html.twig’ %}
where delete_form is the name of the form variable in the twig file.

Examples:

a) Add a class to the form (paste the following into your fields.html.twig file):

# line 270 in base file
{%- block form_start -%}
    {% set method = method|upper %}
    {%- if method in ["GET", "POST"] -%}
        {% set form_method = method %}
    {%- else -%}
        {% set form_method = "POST" %}
    {%- endif -%}
    <form name="{{ name }}" method="{{ form_method|lower }}"
{% if action != '' %} action="{{ action }}"{% endif %}
{% for attrname, attrvalue in attr %} {{ attrname }}="{{ attrvalue }}"
{% endfor %}{% if multipart %} enctype="multipart/form-data"{% endif %} 
class="form-class-name">
    {%- if form_method != method -%}
        <input type="hidden" name="_method" value="{{ method }}" />
    {%- endif -%}
{%- endblock form_start -%}

b) Add class to basic text fields:

{%- block form_widget_simple -%}
    {%- set type = type|default('text') -%}
    <input type="{{ type }}" class="input-class-name" {{ block('widget_attributes') }} 
{% if value is not empty %}value="{{ value }}" 
{% endif %}/>
{%- endblock form_widget_simple -%}

Symfony Docs here: http://symfony.com/doc/current/book/forms.html#global-form-theming

How to view all defined routes in Symfony 2 PHP application

Every so often it is useful to list all of the defined routes in your Symfony application, whether to find out the name of the route, or to debug why a route is being used unexpectedly (routes are called based on the first one that matches). To view the routes, simply type the following in the console:

php app/console debug:router

or

php bin/console debug:router

depending on your version.

Resources:
http://symfony.com/doc/current/book/routing.html#visualizing-debugging-routes

Create a simple CRUD app with Symfony2

Symfony2 has an in built command for building simple CRUD pages for your entities. You can use this to create simple admin pages very quickly.

The command is: php app/console generate:doctrine:crud

This will run in interactive mode giving you options on which entity to create the pages for and which pages to create (All the CRUD pages, or just the read pages).

The Configuration format option specifies (amoung other things) where the routing information is stored. I prefer to use yml so that all of my routing information is together.

Resources:

http://symfony.com/doc/current/bundles/SensioGeneratorBundle/commands/generate_doctrine_crud.html

Improved Recommendation Engine

I have been working on an improved recommendation engine for Twitter users you should follow, however the basic concept is easily used for other recommendations.

Simple Algorithm

While there are many different ways to come up with recommendations, a common one is to look at the the most common friends-of-friends that you have.

This algorithm is trivial to implement, in SQL you would use something like:

SELECT f2.follows_user_id, COUNT( * ) 
FROM following AS f1
INNER JOIN following AS f2 ON f1.follows_user_id = f2.user_id
WHERE f1.user_id =16308660
GROUP BY f2.follows_user_id
ORDER BY COUNT( * ) DESC 

and in Chyper you would use something like:

MATCH (n:Member { id:'James'})-[fr:FRIENDS_WITH]->(f)-[fofr:FRIENDS_WITH]->(fof) 
RETURN fof.name, count(fofr) 
ORDER BY count(fofr) DESC

Drawbacks

The algorithm is a very broad brush approach and ignores the fact that within the community of people that I follow, there are clusters of users that might have the same interests, such as Neo4j, Web Development, Gardening or just all work for the same company.

While this broad brush approach is useful to find the most popular people in the network globally, such as Stephen Fry or Richard Branson , it will miss people who are popular within small clusters, but not globally, whom I might also be interested in following.

Simple Social Network

In the example above, Alice and Bob form a cluster (as they follow each other), and they both follow S. This means I am probably interested in following S, even though globally they would rank quite low (below Q and R).

Improving the Algorithm

What I wanted was a way to find the users I should follow within these clusters. First I needed to cluster the people that I follow, you can see how I did this in my post here.

Once I had the different clusters, I updated the algorithm to the following:
1) Find the users in each cluster
2) Find the most popular friends-of-friends in each cluster
3) Rank these users by the percentage of followers in the cluster and the size of the cluster.

In SQL this looks something like

SELECT c.cluster_id, f.follows_user_id, c.cluster_size, count(*) as c, count(*)/c.cluster_size as p
FROM cluster as c
RIGHT JOIN cluster_member as cm ON c.cluster_id = cm.cluster_id
RIGHT JOIN following as f ON f.user_id = cm.member
GROUP BY c.cluster_id, f.follows_user_id
ORDER BY count(*) / cluster_size DESC, `c`.`cluster_size` DESC, count(*) DESC

However that SQL is very inefficant so I used a modified version below;

REATE TEMPORARY TABLE IF NOT EXISTS cluster_method ENGINE=MEMORY 
as (
SELECT c.cluster_id, f.follows_user_id, c.cluster_size, count(*) as c, count(*)/c.cluster_size as p
FROM cluster as c
RIGHT JOIN cluster_member as cm ON c.cluster_id = cm.cluster_id
RIGHT JOIN following as f ON f.user_id = cm.member
WHERE c.cluster_size >= 5
GROUP BY c.cluster_id, f.follows_user_id
HAVING count(*) / cluster_size > 0.6
ORDER BY `c`.`cluster_size` DESC, count(*) DESC);

DELETE FROM cluster_method WHERE follows_user_id IN
(SELECT follows_user_id FROM following WHERE user_id = $twitterUserId);

SELECT follows_user_id as id, MAX(p) as p
FROM cluster_method 
WHERE follows_user_id <> 16308660 
GROUP BY follows_user_id
ORDER BY MAX(p) DESC, cluster_size DESC;

In Chypher it is much simpler and looks like this:

MATCH (n:Member { id:'James'})-[cr:MEMBER_OF]->(c)<-[cr1:MEMBER_OF]-(f)-[fofr:FRIENDS_WITH]->(fof) 
RETURN c.name, fof.name, count(cr1), count(fofr) 
ORDER BY count(fofr)/count(cr1) DESC, count(cr1) DESC

This new algorithm is not perfect, one issue is that with smaller cluster it is easier for every member to follow a particular user do potentially the ranking needs to be tweaked to account for this. One thing I did in the production implementation was ignore sub-communities with a size less that 5. You also need to do the extra step of determining the sub-communities up front, however even with these issues this algorithm helped me find interesting people to follow that I would otherwise not have found such as the awesome Nanogirl

Note: In both examples I haven’t accounted for users that you already follow, you can remove these easily once you have the list of recommended users to follow. I didn’t show it in the examples above as the method is the same in both cases and I wanted to make the examples clearer.

Grouping Twitter Users based on Followers

For an upcoming project I needed to group the people I follow into communities. While there are many algorithms available for grouping nodes in a graph (in this case a node is a user and an edge represents a follow), they didn’t work for me for 3 reasons:

  1. My graph is directed
  2. I wanted users to be able to be part of more than one community
  3. I needed to be able to cluster quickly and with limited memory

After much experimentation, I did eventually come up with a reasonably good method, which is a modified form of hierarchical clustering. While there is room for improvement, it does find the main communities which is what I needed.

In this post I will use mostly psuedo-code, but you can download a working implementation from GitHub.

The algorithm is as follows:

  1. Load users and who they follow into an array with the following structure:
    following[userId] = array(followsUserId1, followsUserId2, followsUserId3);
    


  2. Invert the array so that users are listed with the people who follow them. This is so that we can find people who are similar because the same people follow them. The reason for this is that 2 people might follow the same users, but those users they follow are from different communities e.g. a tech person and a sports person. But the people who are following a person are usually following someone because they are members of a particular community.
    This also solves a secondary problem where lots of people might follow someone because they are an important member of a community, but that person may not follow very many people.

    follows[userId] = array(followedByUserId1, followedByUserId2, followedByUserId3);
    


  3. Next is the first of two loops (in the code on GitHub you can skip the second loop). This loop creates the initial clusters by doing the following:
    for each user
        find the closest user based on common followers
        use these 2 users to seed a cluster
        set StillAddingUsers = true
        while StillAddingUsers
            find the closest user that is not in the cluster 
            if closest user is close enough
              add user to cluster
              recalculate the common followers of the cluster
            else
              set StillAddingUsers = false  
    


  4. Now that we have a collection of clusters, I then do a merge step. The reason for this as the above algorithm will produce several clusters that are very similar to each other. For example id User1 and User2 are similar, then you will have a cluster from when we looped over User1 and another from when we looped over User2.
    set StillMergingClusters = true
    while StillMergingClusters
       StillMergingClusters = false //to prevent constant looping once we are done
       for each cluster
           find cluster with highest number of common users
           if number common users > minimum value (set in config)
               merge these clusters
               reset for each loop
    


  5. The final step is to store the clusters in the database so that we can refer back to them later.

As I mentioned at the start of this post, this algorithm is not perfect, but was good enough for what I needed. If you have any suggestions for improvement, drop me a line on twitter or submit a PR on GitHub.

Code example is available on GitHub.

TradeMe

TradeMe Homepage

Website: http://trademe.co.nz/

Description:
TradeMe is New Zealand’s largest website with over 3.6 million users in a country of 4.4 million. It is an online auction and classifieds platform covering General Goods, Motors, Property and Jobs.

My role covered both bug fixes as well and new developments including and upload function for Job CV’s and a new selling process for the Motors section of the website.

Technologies:

.Net, SQL Server, Javascript

Example of using Markov Clustering in PHP

This post will go step by step through using my Markov Clustering (MCL) php library to cluster twitter users. For more information on the library see my post here.

You can download the library from GitHib at https://github.com/jrowlandsnz/mcl-php.

For this example we are going to cluster the following social graph from twitter (but this could easily be any social graph). We are also going to assume that all the follows are reciprocal.

 

twitter_mcl_example

 

You can see there are two obvious clusters, but there is one connection between them. The test will be to make sure we finish with two clusters, one of Alice, Bridget, Charles and Doug. The second will be Mark, Nancy and Oliver.

The data is stored in a MySQL database with the following structure:

user
user_id
screen_name
follows
user_id
follows_user_id

All sample code is available on github. The steps involved to use the MCL class are outlines below:

  1. Include the Matrix and MCL classes
    include_once('../Matrix.php');
    include_once('../MCL.php');
    
  2. Connect to Database

    //connect to database
    $dbServer = 'localhost';
    $dbUser = 'test_user';
    $dbPassword = 'test_password';
    $dbName = 'twitter';
    
    $dbConn = mysqli_connect($dbServer, $dbUser, $dbPassword, $dbName);
    if (mysqli_connect_errno()) {
        echo "Unable to connect to the database server at this time";
        exit();
    }
    
  3. Declare some variables to hold our data

    $matrixKey = array();
    $dbData = array();
    $filePrefix = 'twitter_test_data'; //optional, used to output data files
    
  4. Query the database

    This query is very simple, for each of the rows in the following table, join on the user table to get the screen name. You can skip this step if you want to use the id’s and then do the lookup later. I prefer to use the screen_names so the clusters are more readable at the end.

    $sql = "SELECT u1.screen_name as user_name, u2.screen_name as follows_user_name
            FROM `following` f
    	INNER JOIN user u1 ON u1.user_id = f.user_id
    	INNER JOIN user u2 ON u2.user_id = f.follows_user_id";
    
  5. Pull the data and create an array of matrix keys.

    We will build an array of all of the screen_names so we can use these as the row/column labels in the matrix, this will make the matrix much easier to interpret. If we didn’t do this we would have no way of correlating a row or column in the matrix to a specific user.

    while($row = $result->fetch_assoc()) {
    	//get a list of all the keys that will be in the array
    	$key1 = array_search("".$row['user_name'],$matrixKey,TRUE);
    	if($key1 === false) {
    		$matrixKey[] = "".$row['user_name'];
    	}
    	
    	$key2 = array_search("".$row['follows_user_name'],$matrixKey,TRUE);
    	if($key2 === false) {
    		$matrixKey[] = "".$row['follows_user_name'];
    	}
    		
                    //store the data so we don't need to query the db again
    		$dbData[] = $row; 
    	}
    	asort($matrixKey);
    }
    
  6. Now that we know how big our matrix has to be, we will create the matrix and then fill with zeroes so that we know each element has a value and we don’t need to check later. This is more efficient to do than checking if every element has a value on each of the multiplication’s.

    matrixData = array();
    for($i = 0;  $i < sizeof($matrixKey); $i++) {
    		
    	$rowOfZero = array();
    	for($j = 0;  $j < sizeof($matrixKey); $j++) {		
    		$rowOfZero[] = 0;
    	}
    	$matrixData[] = $rowOfZero;
    }
    
  7. We can now load the data of who follows who into the matrix. I am going to use an array of the database data that I created earlier, but you could query the database again if you need to.

    $matrix = new Matrix($matrixData);
    		
    //now populate with data from database
    foreach($dbData as $row) {
    	$rowIndex = array_search("".$row['user_name'], $matrixKey, TRUE);
    	if($rowIndex === FALSE) {
    		throw new Exception("Matrix Key not present");
    	} 
    	
    	$colIndex = array_search("".$row['follows_user_name'], $matrixKey, TRUE);
    	if($colIndex === FALSE) {
    		throw new Exception("Matrix Key not present");
    	} 
    	//need to convert from array index to matrix index 		
            $rowIndex = $rowIndex + 1; 
    	$colIndex = $colIndex + 1;		
    	$matrix->setElement($rowIndex, $colIndex, 1);
    }
    
  8. We can finally actually do the clustering, which is nice and easy.

    The inflation values will make strong connections stronger and weaker connections weaker.
    The power value determines the amount of flow between different cluster.
    You should try different values to determine which works best for your data.

    $mcl = new MCL($matrix);
    $mcl->dataFilePrefix = $filePrefix;
    $mcl->matrixKeyArray = $matrixKey;
    $mcl->inflationValue = 2;
    $mcl->powerValue = 2;
    $mcl->cluster(true);
    $clusters = $mcl->interpret();	
    
  9. Finally we can view the output:

    //print the clusters to the screen
    echo '<pre>';
    print_r($clusters);
    echo '</pre>';
    

    which will display as

    Array
    (
        [0] => Array
            (
                [0] => Alice
                [1] => Doug
                [2] => Bridget
                [3] => Charles
            )
    
        [1] => Array
            (
                [0] => Mark
                [1] => Nancy
                [2] => Oliver
            )
    
    )
    

    Exactly as we expected.

I hope that you found this useful, contact me on twitter, I am @jrowlands and let me know how it went.

All code is available at https://github.com/jrowlandsnz/mcl-php.