Predicting fatalities in car crashes, a recommendation system

FARS_Recommendation_System

This project was done as part of Data Mining course from the Master of Science in Data Science at SMU

Team Members:

  • Lisa Mendez
  • Brandon Lawrence
  • Dominique Edwards
  • Mariana Llamas-Cendon

For this task we decided to test if weather can be an important factor to predict the number of fatalities in each U.S. state.

We are using STATE as the user_id, then there will be only 50 users.

In [2]:
import graphlab as gl

#import data and create an SFrame
df_accident = gl.SFrame.read_csv('accident_good.csv')

#create an object for the model
model = gl.recommender.create(df_accident, user_id="STATE", item_id="WEATHER", target="FATALS")
results = model.recommend(users=None, k=5)
model.save("my_model")
Finished parsing file /Users/marianallamas-cendon/Documents/Portfolio/accident_good.csv
Parsing completed. Parsed 100 lines in 0.198034 secs.
------------------------------------------------------
Inferred types from first 100 line(s) of file as 
column_type_hints=[int,int,int,int,int,int,int,int,int,int,int,int,int,int,int,int,int,int,int,int,int]
If parsing fails due to incorrect types, you can correct
the inferred type list above and pass it to read_csv in
the column_type_hints argument
------------------------------------------------------
Finished parsing file /Users/marianallamas-cendon/Documents/Portfolio/accident_good.csv
Parsing completed. Parsed 28823 lines in 0.134193 secs.
Recsys training: model = ranking_factorization_recommender
Preparing data set.
    Data has 28823 observations with 50 users and 9 items.
    Data prepared in: 0.224137s
Training ranking_factorization_recommender for recommendations.
+--------------------------------+--------------------------------------------------+----------+
| Parameter                      | Description                                      | Value    |
+--------------------------------+--------------------------------------------------+----------+
| num_factors                    | Factor Dimension                                 | 32       |
| regularization                 | L2 Regularization on Factors                     | 1e-09    |
| solver                         | Solver used for training                         | adagrad  |
| linear_regularization          | L2 Regularization on Linear Coefficients         | 1e-09    |
| ranking_regularization         | Rank-based Regularization Weight                 | 0.25     |
| max_iterations                 | Maximum Number of Iterations                     | 25       |
+--------------------------------+--------------------------------------------------+----------+
  Optimizing model using SGD; tuning step size.
  Using 10000 / 28823 points for tuning the step size.
+---------+-------------------+------------------------------------------+
| Attempt | Initial Step Size | Estimated Objective Value                |
+---------+-------------------+------------------------------------------+
| 0       | 2.5               | Not Viable                               |
| 1       | 0.625             | Not Viable                               |
| 2       | 0.15625           | Not Viable                               |
| 3       | 0.0390625         | No Decrease (0.314831 >= 0.237784)       |
| 4       | 0.00976562        | 0.201801                                 |
| 5       | 0.00488281        | 0.219325                                 |
| 6       | 0.00244141        | 0.221194                                 |
| 7       | 0.0012207         | 0.225937                                 |
+---------+-------------------+------------------------------------------+
| Final   | 0.00976562        | 0.201801                                 |
+---------+-------------------+------------------------------------------+
Starting Optimization.
+---------+--------------+-------------------+-----------------------+-------------+
| Iter.   | Elapsed Time | Approx. Objective | Approx. Training RMSE | Step Size   |
+---------+--------------+-------------------+-----------------------+-------------+
| Initial | 105us        | 0.242637          | 0.351786              |             |
+---------+--------------+-------------------+-----------------------+-------------+
| 1       | 381.249ms    | 0.116904          | 0.341146              | 0.00976562  |
| 2       | 692.135ms    | 0.114628          | 0.33784               | 0.00976562  |
| 3       | 1.05s        | 0.114367          | 0.337454              | 0.00976562  |
| 4       | 1.40s        | 0.114558          | 0.337749              | 0.00976562  |
| 5       | 1.72s        | 0.114477          | 0.337626              | 0.00976562  |
| 6       | 2.08s        | 0.114643          | 0.337881              | 0.00976562  |
| 10      | 3.45s        | 0.115435          | 0.339049              | 0.00976562  |
| 11      | 3.76s        | 0.115798          | 0.339584              | 0.00976562  |
| 15      | 4.92s        | 0.116656          | 0.340852              | 0.00976562  |
| 20      | 6.21s        | 0.11776           | 0.342467              | 0.00976562  |
| 25      | 7.50s        | 0.118948          | 0.34419               | 0.00976562  |
+---------+--------------+-------------------+-----------------------+-------------+
Optimization Complete: Maximum number of passes through the data reached.
Computing final objective value and training RMSE.
       Final objective value: 0.219613
       Final training RMSE: 0.343735
In [3]:
results
Out[3]:
STATE WEATHER score rank
1 3 1.15666897176 1
1 11 0.989689914046 2
1 12 0.980822182262 3
1 6 0.980543899817 4
2 3 1.13302302128 1
2 11 0.98654121021 2
2 6 0.979254902808 3
4 3 1.08496082964 1
4 11 0.978721411609 2
4 12 0.974025529986 3

[150 rows x 4 columns]
Note: Only the head of the SFrame is printed.
You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.

In the above table, we can see three of the states showing: 1 (Alabama), 2 (Alaska) and 4 (Arizona). It is evident that from the nine levels of the attribute WEATHER conditions 3 (sleet, hail), 6 (Severe crosswinds), 11 (blowing snow), and 12 (freezing rain or drizzle) are the most preponderant. The three states shown display the highest scores when sleet or hail occurs.

Item-item recommendation

In [4]:
# from graphlab.recommender import item_similarity_recommender

item_item = gl.recommender.item_similarity_recommender.create(df_accident, 
                                  user_id="STATE", 
                                  item_id="WEATHER", 
                                  target="FATALS",
                                  only_top_k=3,
                                  similarity_type="cosine")

results = item_item.get_similar_items(k=3)
results.head()
Recsys training: model = item_similarity
Warning: Ignoring columns , VE_TOTAL, PVH_INVL, PEDS, PERNOTMVIT, PERMVIT, PERSONS, DAY_WEEK, ROUTE, MAN_COLL, RELJCT1, RELJCT2, WRK_ZONE, REL_ROAD, LGT_COND, DRUNK_DR, HARM_EV_RED, HR_RANGE;
    To use these columns in scoring predictions, use a model that allows the use of additional features.
Preparing data set.
    Data has 28823 observations with 50 users and 9 items.
    Data prepared in: 0.082551s
Training model from provided data.
Gathering per-item and per-user statistics.
+--------------------------------+------------+
| Elapsed Time (Item Statistics) | % Complete |
+--------------------------------+------------+
| 397us                          | 100        |
+--------------------------------+------------+
Setting up lookup tables.
Processing data in one pass using dense lookup tables.
+-------------------------------------+------------------+-----------------+
| Elapsed Time (Constructing Lookups) | Total % Complete | Items Processed |
+-------------------------------------+------------------+-----------------+
| 782us                               | 1                | 0               |
| 1.055ms                             | 100              | 9               |
+-------------------------------------+------------------+-----------------+
Finalizing lookup tables.
Generating candidate set for working with new users.
Finished training in 0.017355s
Out[4]:
WEATHER similar score rank
1 2 0.986107587814 1
1 10 0.950036883354 2
1 5 0.921882271767 3
10 2 0.956637382507 1
10 1 0.950036883354 2
10 5 0.890565276146 3
2 1 0.986107587814 1
2 10 0.956637382507 2
2 5 0.924809992313 3
5 2 0.924809992313 1

[10 rows x 4 columns]

In this case it is interesting that the item-item matrix is displaying WEATHER 1 (clear) as similar to WEATHER 2 (rain), WEATHER 10 (cloudy), WEATHER 5 (fog, smog, smoke).

Divide the data into train (80%) and test (20%). And we will use root mean square (RMSE) as the measure.

In [5]:
train, test = gl.recommender.util.random_split_by_user(df_accident,
                                                    user_id="STATE", item_id="WEATHER",
                                                    max_num_users=100, item_test_proportion=0.2)

Our data is densely populated, so using cosine is not the best option. But Graphlab only provides three metrics for similarity: Cosine, Jaccard and Pearson. So we decided to use Pearson because it measures the correlation in relation to what we are trying to predict.

In [6]:
from IPython.display import display
from IPython.display import Image

gl.canvas.set_target('ipynb')


item_item = gl.recommender.item_similarity_recommender.create(train, 
                                  user_id="STATE", 
                                  item_id="WEATHER", 
                                  target="FATALS",
                                  only_top_k=5,
                                  similarity_type="pearson")

rmse_results = item_item.evaluate(test)
Recsys training: model = item_similarity
Warning: Ignoring columns , VE_TOTAL, PVH_INVL, PEDS, PERNOTMVIT, PERMVIT, PERSONS, DAY_WEEK, ROUTE, MAN_COLL, RELJCT1, RELJCT2, WRK_ZONE, REL_ROAD, LGT_COND, DRUNK_DR, HARM_EV_RED, HR_RANGE;
    To use these columns in scoring predictions, use a model that allows the use of additional features.
Preparing data set.
    Data has 23035 observations with 50 users and 9 items.
    Data prepared in: 0.027989s
Training model from provided data.
Gathering per-item and per-user statistics.
+--------------------------------+------------+
| Elapsed Time (Item Statistics) | % Complete |
+--------------------------------+------------+
| 18.888ms                       | 100        |
+--------------------------------+------------+
Setting up lookup tables.
Processing data in one pass using dense lookup tables.
+-------------------------------------+------------------+-----------------+
| Elapsed Time (Constructing Lookups) | Total % Complete | Items Processed |
+-------------------------------------+------------------+-----------------+
| 19.414ms                            | 0                | 0               |
| 19.951ms                            | 100              | 9               |
+-------------------------------------+------------------+-----------------+
Finalizing lookup tables.
Generating candidate set for working with new users.
Finished training in 0.030628s
Precision and recall summary statistics by cutoff
+--------+-----------------+------------------+
| cutoff |  mean_precision |   mean_recall    |
+--------+-----------------+------------------+
|   1    |       0.04      | 0.00021666757627 |
|   2    |       0.04      | 0.00080480111718 |
|   3    |       0.06      | 0.00524961579096 |
|   4    |      0.055      | 0.00585567639702 |
|   5    |      0.052      | 0.00585567639702 |
|   6    | 0.0513333333333 | 0.00585567639702 |
|   7    | 0.0513333333333 | 0.00585567639702 |
|   8    | 0.0513333333333 | 0.00585567639702 |
|   9    | 0.0513333333333 | 0.00585567639702 |
|   10   | 0.0513333333333 | 0.00585567639702 |
+--------+-----------------+------------------+
[10 rows x 3 columns]

('\nOverall RMSE: ', 0.35860930544969194)

Per User RMSE (best)
+-------+-------+-----------------+
| STATE | count |       rmse      |
+-------+-------+-----------------+
|   23  |   25  | 0.0799602579595 |
+-------+-------+-----------------+
[1 rows x 3 columns]


Per User RMSE (worst)
+-------+-------+----------------+
| STATE | count |      rmse      |
+-------+-------+----------------+
|   19  |   51  | 0.654436284432 |
+-------+-------+----------------+
[1 rows x 3 columns]


Per Item RMSE (best)
+---------+-------+----------------+
| WEATHER | count |      rmse      |
+---------+-------+----------------+
|    11   |   2   | 0.044691629257 |
+---------+-------+----------------+
[1 rows x 3 columns]


Per Item RMSE (worst)
+---------+-------+----------------+
| WEATHER | count |      rmse      |
+---------+-------+----------------+
|    4    |   65  | 0.404771824069 |
+---------+-------+----------------+
[1 rows x 3 columns]

In [7]:
print rmse_results.viewkeys()
dict_keys(['rmse_by_user', 'precision_recall_overall', 'rmse_by_item', 'precision_recall_by_user', 'rmse_overall'])
In [8]:
rmse_results['rmse_by_user']
Out[8]:
STATE count rmse
49 49 0.24051573004
13 261 0.321557134115
36 210 0.33407896615
24 86 0.463339273551
2 13 0.268369795182
46 24 0.336526581733
30 32 0.291987148142
53 97 0.503115623437
42 225 0.352487467897
44 9 0.0917102727182

[50 rows x 3 columns]
Note: Only the head of the SFrame is printed.
You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.

States with lower counts display lower RMSE.

In [9]:
from matplotlib import pyplot as plt
%matplotlib inline
plt.style.use('ggplot')

rmsevals = rmse_results['rmse_by_user']['rmse']
rmsevals = rmsevals[rmse_results['rmse_by_user']['count']>50]

plt.hist(rmsevals,bins=20)
plt.title('RMSE by User')
plt.show()
In [10]:
rmse_results['precision_recall_by_user']
Out[10]:
STATE cutoff precision recall count
1 1 0.0 0.0 148
1 2 0.0 0.0 148
1 3 0.0 0.0 148
1 4 0.0 0.0 148
1 5 0.0 0.0 148
1 6 0.0 0.0 148
1 7 0.0 0.0 148
1 8 0.0 0.0 148
1 9 0.0 0.0 148
1 10 0.0 0.0 148

[900 rows x 5 columns]
Note: Only the head of the SFrame is printed.
You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.

We suppose that our precision and recall are equal to 0.0 because even though there are states that have higher counts, not all of them correspond to the same count regarding weather conditions.

In [11]:
import graphlab.aggregate as agg

# we will be using these aggregations
agg_list = [agg.AVG('precision'),agg.STD('precision'),agg.AVG('recall'),agg.STD('recall')]

# apply these functions to each group (we will group the results by 'k' which is the cutoff)
# the cutoff is the number of top items to look for see the following URL for the actual equation
# https://dato.com/products/create/docs/generated/graphlab.recommender.util.precision_recall_by_user.html#graphlab.recommender.util.precision_recall_by_user
rmse_results['precision_recall_by_user'].groupby('cutoff',agg_list)

# the groups are not sorted
Out[11]:
cutoff Avg of precision Stdv of precision Avg of recall Stdv of recall
36 0.0513333333333 0.105769140637 0.00585567639702 0.0168259490442
2 0.04 0.135646599663 0.00080480111718 0.00331418885208
46 0.0513333333333 0.105769140637 0.00585567639702 0.0168259490442
31 0.0513333333333 0.105769140637 0.00585567639702 0.0168259490442
26 0.0513333333333 0.105769140637 0.00585567639702 0.0168259490442
8 0.0513333333333 0.105769140637 0.00585567639702 0.0168259490442
5 0.052 0.106595809799 0.00585567639702 0.0168259490442
16 0.0513333333333 0.105769140637 0.00585567639702 0.0168259490442
41 0.0513333333333 0.105769140637 0.00585567639702 0.0168259490442
4 0.055 0.111317863197 0.00585567639702 0.0168259490442

[18 rows x 5 columns]
Note: Only the head of the SFrame is printed.
You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.

In [12]:
rec1 = gl.recommender.ranking_factorization_recommender.create(train, 
                                  user_id="STATE", 
                                  item_id="WEATHER", 
                                  target="FATALS")

rmse_results = rec1.evaluate(test)
Recsys training: model = ranking_factorization_recommender
Preparing data set.
    Data has 23035 observations with 50 users and 9 items.
    Data prepared in: 0.176914s
Training ranking_factorization_recommender for recommendations.
+--------------------------------+--------------------------------------------------+----------+
| Parameter                      | Description                                      | Value    |
+--------------------------------+--------------------------------------------------+----------+
| num_factors                    | Factor Dimension                                 | 32       |
| regularization                 | L2 Regularization on Factors                     | 1e-09    |
| solver                         | Solver used for training                         | adagrad  |
| linear_regularization          | L2 Regularization on Linear Coefficients         | 1e-09    |
| ranking_regularization         | Rank-based Regularization Weight                 | 0.25     |
| max_iterations                 | Maximum Number of Iterations                     | 25       |
+--------------------------------+--------------------------------------------------+----------+
  Optimizing model using SGD; tuning step size.
  Using 10000 / 23035 points for tuning the step size.
+---------+-------------------+------------------------------------------+
| Attempt | Initial Step Size | Estimated Objective Value                |
+---------+-------------------+------------------------------------------+
| 0       | 2.5               | Not Viable                               |
| 1       | 0.625             | Not Viable                               |
| 2       | 0.15625           | Not Viable                               |
| 3       | 0.0390625         | 0.228855                                 |
| 4       | 0.0195312         | 0.233332                                 |
| 5       | 0.00976562        | 0.207959                                 |
| 6       | 0.00488281        | 0.216312                                 |
| 7       | 0.00244141        | 0.223918                                 |
| 8       | 0.0012207         | 0.226384                                 |
+---------+-------------------+------------------------------------------+
| Final   | 0.00976562        | 0.207959                                 |
+---------+-------------------+------------------------------------------+
Starting Optimization.
+---------+--------------+-------------------+-----------------------+-------------+
| Iter.   | Elapsed Time | Approx. Objective | Approx. Training RMSE | Step Size   |
+---------+--------------+-------------------+-----------------------+-------------+
| Initial | 480us        | 0.240382          | 0.350144              |             |
+---------+--------------+-------------------+-----------------------+-------------+
| 1       | 288.674ms    | 0.131897          | 0.362073              | 0.00976562  |
| 2       | 836.489ms    | 0.124734          | 0.35214               | 0.00976562  |
| 3       | 1.13s        | 0.125965          | 0.353851              | 0.00976562  |
| 4       | 1.37s        | 0.125348          | 0.35299               | 0.00976562  |
| 5       | 1.60s        | 0.123566          | 0.350498              | 0.00976562  |
| 6       | 1.83s        | 0.126547          | 0.354712              | 0.00976562  |
| 10      | 2.80s        | 0.127989          | 0.356712              | 0.00976562  |
| 11      | 3.02s        | 0.127397          | 0.355883              | 0.00976562  |
| 15      | 3.85s        | 0.128963          | 0.35806               | 0.00976562  |
| 20      | 4.80s        | 0.130124          | 0.359691              | 0.00976562  |
| 25      | 5.73s        | 0.131749          | 0.361912              | 0.00976562  |
+---------+--------------+-------------------+-----------------------+-------------+
Optimization Complete: Maximum number of passes through the data reached.
Computing final objective value and training RMSE.
       Final objective value: 0.229645
       Final training RMSE: 0.360448
Precision and recall summary statistics by cutoff
+--------+-----------------+------------------+
| cutoff |  mean_precision |   mean_recall    |
+--------+-----------------+------------------+
|   1    |       0.08      | 0.00344741110604 |
|   2    |       0.06      | 0.00428960968761 |
|   3    |       0.06      | 0.00565765659504 |
|   4    |       0.05      | 0.00565765659504 |
|   5    |      0.052      | 0.00585567639702 |
|   6    | 0.0513333333333 | 0.00585567639702 |
|   7    | 0.0513333333333 | 0.00585567639702 |
|   8    | 0.0513333333333 | 0.00585567639702 |
|   9    | 0.0513333333333 | 0.00585567639702 |
|   10   | 0.0513333333333 | 0.00585567639702 |
+--------+-----------------+------------------+
[10 rows x 3 columns]

('\nOverall RMSE: ', 0.3772617563604375)

Per User RMSE (best)
+-------+-------+-----------------+
| STATE | count |       rmse      |
+-------+-------+-----------------+
|   50  |   12  | 0.0783877777429 |
+-------+-------+-----------------+
[1 rows x 3 columns]


Per User RMSE (worst)
+-------+-------+----------------+
| STATE | count |      rmse      |
+-------+-------+----------------+
|   44  |   9   | 0.762158633873 |
+-------+-------+----------------+
[1 rows x 3 columns]


Per Item RMSE (best)
+---------+-------+----------------+
| WEATHER | count |      rmse      |
+---------+-------+----------------+
|    6    |   5   | 0.125193635683 |
+---------+-------+----------------+
[1 rows x 3 columns]


Per Item RMSE (worst)
+---------+-------+----------------+
| WEATHER | count |      rmse      |
+---------+-------+----------------+
|    1    |  4157 | 0.382395837274 |
+---------+-------+----------------+
[1 rows x 3 columns]

In [13]:
rmse_results['precision_recall_by_user'].groupby('cutoff',[agg.AVG('precision'),agg.STD('precision'),agg.AVG('recall'),agg.STD('recall')])
Out[13]:
cutoff Avg of precision Stdv of precision Avg of recall Stdv of recall
36 0.0513333333333 0.105769140637 0.00585567639702 0.0168259490442
2 0.06 0.162480768093 0.00428960968761 0.0160687640684
46 0.0513333333333 0.105769140637 0.00585567639702 0.0168259490442
31 0.0513333333333 0.105769140637 0.00585567639702 0.0168259490442
26 0.0513333333333 0.105769140637 0.00585567639702 0.0168259490442
8 0.0513333333333 0.105769140637 0.00585567639702 0.0168259490442
5 0.052 0.106595809799 0.00585567639702 0.0168259490442
16 0.0513333333333 0.105769140637 0.00585567639702 0.0168259490442
41 0.0513333333333 0.105769140637 0.00585567639702 0.0168259490442
4 0.05 0.108012344973 0.00565765659504 0.016835433973

[18 rows x 5 columns]
Note: Only the head of the SFrame is printed.
You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.

In [14]:
rec1 = gl.recommender.ranking_factorization_recommender.create(train, 
                                  user_id="STATE", 
                                  item_id="WEATHER", 
                                  target="FATALS",
                                  num_factors=8,                 # override the default value
                                  regularization=1e-02,           # override the default value
                                  linear_regularization = 1e-3)   # override the default value

rmse_results = rec1.evaluate(test)
Recsys training: model = ranking_factorization_recommender
Preparing data set.
    Data has 23035 observations with 50 users and 9 items.
    Data prepared in: 0.159856s
Training ranking_factorization_recommender for recommendations.
+--------------------------------+--------------------------------------------------+----------+
| Parameter                      | Description                                      | Value    |
+--------------------------------+--------------------------------------------------+----------+
| num_factors                    | Factor Dimension                                 | 8        |
| regularization                 | L2 Regularization on Factors                     | 0.01     |
| solver                         | Solver used for training                         | adagrad  |
| linear_regularization          | L2 Regularization on Linear Coefficients         | 0.001    |
| ranking_regularization         | Rank-based Regularization Weight                 | 0.25     |
| max_iterations                 | Maximum Number of Iterations                     | 25       |
+--------------------------------+--------------------------------------------------+----------+
  Optimizing model using SGD; tuning step size.
  Using 10000 / 23035 points for tuning the step size.
+---------+-------------------+------------------------------------------+
| Attempt | Initial Step Size | Estimated Objective Value                |
+---------+-------------------+------------------------------------------+
| 0       | 0.319362          | Not Viable                               |
| 1       | 0.0798405         | 0.189069                                 |
| 2       | 0.0399202         | 0.188943                                 |
| 3       | 0.0199601         | 0.194279                                 |
| 4       | 0.00998006        | 0.207271                                 |
| 5       | 0.00499003        | 0.217332                                 |
+---------+-------------------+------------------------------------------+
| Final   | 0.0399202         | 0.188943                                 |
+---------+-------------------+------------------------------------------+
Starting Optimization.
+---------+--------------+-------------------+-----------------------+-------------+
| Iter.   | Elapsed Time | Approx. Objective | Approx. Training RMSE | Step Size   |
+---------+--------------+-------------------+-----------------------+-------------+
| Initial | 178us        | 0.240378          | 0.35015               |             |
+---------+--------------+-------------------+-----------------------+-------------+
| 1       | 158.605ms    | 0.20789           | 0.452598              | 0.0399202   |
| 2       | 245.308ms    | 0.113236          | 0.332743              | 0.0399202   |
| 3       | 393.519ms    | 0.112719          | 0.332559              | 0.0399202   |
| 4       | 497.428ms    | 0.112254          | 0.33232               | 0.0399202   |
| 5       | 573.334ms    | 0.111979          | 0.332262              | 0.0399202   |
| 6       | 708.174ms    | 0.111759          | 0.33221               | 0.0399202   |
| 10      | 1.02s        | 0.111302          | 0.332136              | 0.0399202   |
| 11      | 1.14s        | 0.111169          | 0.332016              | 0.0399202   |
| 20      | 1.75s        | 0.110893          | 0.331859              | 0.0399202   |
+---------+--------------+-------------------+-----------------------+-------------+
Optimization Complete: Maximum number of passes through the data reached.
Computing final objective value and training RMSE.
       Final objective value: 0.204976
       Final training RMSE: 0.331617
Precision and recall summary statistics by cutoff
+--------+-----------------+------------------+
| cutoff |  mean_precision |   mean_recall    |
+--------+-----------------+------------------+
|   1    |       0.1       | 0.00386407777271 |
|   2    |       0.07      | 0.00442659598898 |
|   3    |       0.06      | 0.00524961579096 |
|   4    |      0.055      | 0.00585567639702 |
|   5    |      0.052      | 0.00585567639702 |
|   6    | 0.0513333333333 | 0.00585567639702 |
|   7    | 0.0513333333333 | 0.00585567639702 |
|   8    | 0.0513333333333 | 0.00585567639702 |
|   9    | 0.0513333333333 | 0.00585567639702 |
|   10   | 0.0513333333333 | 0.00585567639702 |
+--------+-----------------+------------------+
[10 rows x 3 columns]

('\nOverall RMSE: ', 0.3410664467853666)

Per User RMSE (best)
+-------+-------+----------------+
| STATE | count |      rmse      |
+-------+-------+----------------+
|   50  |   12  | 0.125088800612 |
+-------+-------+----------------+
[1 rows x 3 columns]


Per User RMSE (worst)
+-------+-------+----------------+
| STATE | count |      rmse      |
+-------+-------+----------------+
|   19  |   51  | 0.606399940191 |
+-------+-------+----------------+
[1 rows x 3 columns]


Per Item RMSE (best)
+---------+-------+-----------------+
| WEATHER | count |       rmse      |
+---------+-------+-----------------+
|    11   |   2   | 0.0968528209241 |
+---------+-------+-----------------+
[1 rows x 3 columns]


Per Item RMSE (worst)
+---------+-------+----------------+
| WEATHER | count |      rmse      |
+---------+-------+----------------+
|    3    |   18  | 0.412185322729 |
+---------+-------+----------------+
[1 rows x 3 columns]

In [15]:
comparison = gl.recommender.util.compare_models(test, [item_item, rec1])
PROGRESS: Evaluate model M0

Precision and recall summary statistics by cutoff
+--------+-----------------+------------------+
| cutoff |  mean_precision |   mean_recall    |
+--------+-----------------+------------------+
|   1    |       0.04      | 0.00021666757627 |
|   2    |       0.04      | 0.00080480111718 |
|   3    |       0.06      | 0.00524961579096 |
|   4    |      0.055      | 0.00585567639702 |
|   5    |      0.052      | 0.00585567639702 |
|   6    | 0.0513333333333 | 0.00585567639702 |
|   7    | 0.0513333333333 | 0.00585567639702 |
|   8    | 0.0513333333333 | 0.00585567639702 |
|   9    | 0.0513333333333 | 0.00585567639702 |
|   10   | 0.0513333333333 | 0.00585567639702 |
+--------+-----------------+------------------+
[10 rows x 3 columns]

('\nOverall RMSE: ', 0.35860930544969194)

Per User RMSE (best)
+-------+-------+-----------------+
| STATE | count |       rmse      |
+-------+-------+-----------------+
|   23  |   25  | 0.0799602579595 |
+-------+-------+-----------------+
[1 rows x 3 columns]


Per User RMSE (worst)
+-------+-------+----------------+
| STATE | count |      rmse      |
+-------+-------+----------------+
|   19  |   51  | 0.654436284432 |
+-------+-------+----------------+
[1 rows x 3 columns]


Per Item RMSE (best)
+---------+-------+----------------+
| WEATHER | count |      rmse      |
+---------+-------+----------------+
|    11   |   2   | 0.044691629257 |
+---------+-------+----------------+
[1 rows x 3 columns]


Per Item RMSE (worst)
+---------+-------+----------------+
| WEATHER | count |      rmse      |
+---------+-------+----------------+
|    4    |   65  | 0.404771824069 |
+---------+-------+----------------+
[1 rows x 3 columns]

PROGRESS: Evaluate model M1

Precision and recall summary statistics by cutoff
+--------+-----------------+------------------+
| cutoff |  mean_precision |   mean_recall    |
+--------+-----------------+------------------+
|   1    |       0.1       | 0.00386407777271 |
|   2    |       0.07      | 0.00442659598898 |
|   3    |       0.06      | 0.00524961579096 |
|   4    |      0.055      | 0.00585567639702 |
|   5    |      0.052      | 0.00585567639702 |
|   6    | 0.0513333333333 | 0.00585567639702 |
|   7    | 0.0513333333333 | 0.00585567639702 |
|   8    | 0.0513333333333 | 0.00585567639702 |
|   9    | 0.0513333333333 | 0.00585567639702 |
|   10   | 0.0513333333333 | 0.00585567639702 |
+--------+-----------------+------------------+
[10 rows x 3 columns]

('\nOverall RMSE: ', 0.3410664467853666)

Per User RMSE (best)
+-------+-------+----------------+
| STATE | count |      rmse      |
+-------+-------+----------------+
|   50  |   12  | 0.125088800612 |
+-------+-------+----------------+
[1 rows x 3 columns]


Per User RMSE (worst)
+-------+-------+----------------+
| STATE | count |      rmse      |
+-------+-------+----------------+
|   19  |   51  | 0.606399940191 |
+-------+-------+----------------+
[1 rows x 3 columns]


Per Item RMSE (best)
+---------+-------+-----------------+
| WEATHER | count |       rmse      |
+---------+-------+-----------------+
|    11   |   2   | 0.0968528209241 |
+---------+-------+-----------------+
[1 rows x 3 columns]


Per Item RMSE (worst)
+---------+-------+----------------+
| WEATHER | count |      rmse      |
+---------+-------+----------------+
|    3    |   18  | 0.412185322729 |
+---------+-------+----------------+
[1 rows x 3 columns]

In [16]:
comparisonstruct = gl.compare(test,[item_item, rec1])
PROGRESS: Evaluate model M0

Precision and recall summary statistics by cutoff
+--------+-----------------+------------------+
| cutoff |  mean_precision |   mean_recall    |
+--------+-----------------+------------------+
|   1    |       0.04      | 0.00021666757627 |
|   2    |       0.04      | 0.00080480111718 |
|   3    |       0.06      | 0.00524961579096 |
|   4    |      0.055      | 0.00585567639702 |
|   5    |      0.052      | 0.00585567639702 |
|   6    | 0.0513333333333 | 0.00585567639702 |
|   7    | 0.0513333333333 | 0.00585567639702 |
|   8    | 0.0513333333333 | 0.00585567639702 |
|   9    | 0.0513333333333 | 0.00585567639702 |
|   10   | 0.0513333333333 | 0.00585567639702 |
+--------+-----------------+------------------+
[10 rows x 3 columns]

PROGRESS: Evaluate model M1

Precision and recall summary statistics by cutoff
+--------+-----------------+------------------+
| cutoff |  mean_precision |   mean_recall    |
+--------+-----------------+------------------+
|   1    |       0.1       | 0.00386407777271 |
|   2    |       0.07      | 0.00442659598898 |
|   3    |       0.06      | 0.00524961579096 |
|   4    |      0.055      | 0.00585567639702 |
|   5    |      0.052      | 0.00585567639702 |
|   6    | 0.0513333333333 | 0.00585567639702 |
|   7    | 0.0513333333333 | 0.00585567639702 |
|   8    | 0.0513333333333 | 0.00585567639702 |
|   9    | 0.0513333333333 | 0.00585567639702 |
|   10   | 0.0513333333333 | 0.00585567639702 |
+--------+-----------------+------------------+
[10 rows x 3 columns]

Model compare metric: precision_recall
In [17]:
gl.show_comparison(comparisonstruct,[item_item, rec1])

The recommendation seems to provide lower rates for precision and recall than that of the item-item.

In [18]:
params = {'user_id': 'STATE', 
          'item_id': 'WEATHER', 
          'target': 'FATALS',
          'num_factors': [8, 12, 16, 24, 32], 
          'regularization':[0.001] ,
          'linear_regularization': [0.001]}

job = gl.model_parameter_search.create( (train,test),
        gl.recommender.ranking_factorization_recommender.create,
        params,
        max_models=5,
        environment=None)

# also note thatthis evaluator also supports sklearn
# https://dato.com/products/create/docs/generated/graphlab.toolkits.model_parameter_search.create.html?highlight=model_parameter_search
[INFO] graphlab.deploy.job: Validating job.
[INFO] graphlab.deploy.job: Creating a LocalAsync environment called 'async'.
[INFO] graphlab.deploy.map_job: Validation complete. Job: 'Model-Parameter-Search-May-07-2018-14-14-1800000' ready for execution
[INFO] graphlab.deploy.map_job: Job: 'Model-Parameter-Search-May-07-2018-14-14-1800000' scheduled.
[INFO] graphlab.deploy.job: Validating job.
[INFO] graphlab.deploy.map_job: A job with name 'Model-Parameter-Search-May-07-2018-14-14-1800000' already exists. Renaming the job to 'Model-Parameter-Search-May-07-2018-14-14-1800000-5b671'.
[INFO] graphlab.deploy.map_job: Validation complete. Job: 'Model-Parameter-Search-May-07-2018-14-14-1800000-5b671' ready for execution
[INFO] graphlab.deploy.map_job: Job: 'Model-Parameter-Search-May-07-2018-14-14-1800000-5b671' scheduled.
In [19]:
job.get_status()
Out[19]:
{'Canceled': 0, 'Completed': 0, 'Failed': 0, 'Pending': 5, 'Running': 0}
In [20]:
job_result = job.get_results()

job_result.head()
Out[20]:
model_id item_id linear_regularization max_iterations num_factors num_sampled_negative_exam
ples …
ranking_regularization
1 WEATHER 0.001 25 24 4 0.1
0 WEATHER 0.001 50 12 8 0.1
3 WEATHER 0.001 25 12 8 0.1
2 WEATHER 0.001 50 32 8 0.1
4 WEATHER 0.001 25 32 4 0.5
regularization target user_id training_precision@5 training_recall@5 training_rmse validation_precision@5
0.001 FATALS STATE 0.856 0.0219015334853 0.338885532954 0.052
0.001 FATALS STATE 0.856 0.0219015334853 0.332005535795 0.052
0.001 FATALS STATE 0.856 0.0219015334853 0.331303684853 0.052
0.001 FATALS STATE 0.856 0.0219015334853 0.333030975697 0.052
0.001 FATALS STATE 0.916 0.02485691372 0.331198490225 0.052
validation_recall@5 validation_rmse
0.00585567639702 0.35092818692
0.00585567639702 0.342066633074
0.00585567639702 0.341090968019
0.00585567639702 0.344317221565
0.00585567639702 0.341000969847

[5 rows x 16 columns]

In [21]:
bst_prms = job.get_best_params()
bst_prms
Out[21]:
{'item_id': 'WEATHER',
 'linear_regularization': 0.001,
 'max_iterations': 25,
 'num_factors': 32,
 'num_sampled_negative_examples': 4,
 'ranking_regularization': 0.5,
 'regularization': 0.001,
 'target': 'FATALS',
 'user_id': 'STATE'}
In [22]:
models = job.get_models()
models
Out[22]:
[Class                            : RankingFactorizationRecommender
 
 Schema
 ------
 User ID                          : STATE
 Item ID                          : WEATHER
 Target                           : FATALS
 Additional observation features  : 18
 User side features               : []
 Item side features               : []
 
 Statistics
 ----------
 Number of observations           : 23035
 Number of users                  : 50
 Number of items                  : 9
 
 Training summary
 ----------------
 Training time                    : 10.9364
 
 Model Parameters
 ----------------
 Model class                      : RankingFactorizationRecommender
 num_factors                      : 12
 binary_target                    : 0
 side_data_factorization          : 1
 solver                           : auto
 nmf                              : 0
 max_iterations                   : 50
 
 Regularization Settings
 -----------------------
 regularization                   : 0.001
 regularization_type              : normal
 linear_regularization            : 0.001
 ranking_regularization           : 0.1
 unobserved_rating_value          : -1.79769313486e+308
 num_sampled_negative_examples    : 8
 ials_confidence_scaling_type     : auto
 ials_confidence_scaling_factor   : 1
 
 Optimization Settings
 ---------------------
 init_random_sigma                : 0.01
 sgd_convergence_interval         : 4
 sgd_convergence_threshold        : 0.0
 sgd_max_trial_iterations         : 5
 sgd_sampling_block_size          : 131072
 sgd_step_adjustment_interval     : 4
 sgd_step_size                    : 0.0
 sgd_trial_sample_minimum_size    : 10000
 sgd_trial_sample_proportion      : 0.125
 step_size_decrease_rate          : 0.75
 additional_iterations_if_unhealthy : 5
 adagrad_momentum_weighting       : 0.9
 num_tempering_iterations         : 4
 tempering_regularization_start_value : 0.0
 track_exact_loss                 : 0,
 Class                            : RankingFactorizationRecommender
 
 Schema
 ------
 User ID                          : STATE
 Item ID                          : WEATHER
 Target                           : FATALS
 Additional observation features  : 18
 User side features               : []
 Item side features               : []
 
 Statistics
 ----------
 Number of observations           : 23035
 Number of users                  : 50
 Number of items                  : 9
 
 Training summary
 ----------------
 Training time                    : 9.5872
 
 Model Parameters
 ----------------
 Model class                      : RankingFactorizationRecommender
 num_factors                      : 24
 binary_target                    : 0
 side_data_factorization          : 1
 solver                           : auto
 nmf                              : 0
 max_iterations                   : 25
 
 Regularization Settings
 -----------------------
 regularization                   : 0.001
 regularization_type              : normal
 linear_regularization            : 0.001
 ranking_regularization           : 0.1
 unobserved_rating_value          : -1.79769313486e+308
 num_sampled_negative_examples    : 4
 ials_confidence_scaling_type     : auto
 ials_confidence_scaling_factor   : 1
 
 Optimization Settings
 ---------------------
 init_random_sigma                : 0.01
 sgd_convergence_interval         : 4
 sgd_convergence_threshold        : 0.0
 sgd_max_trial_iterations         : 5
 sgd_sampling_block_size          : 131072
 sgd_step_adjustment_interval     : 4
 sgd_step_size                    : 0.0
 sgd_trial_sample_minimum_size    : 10000
 sgd_trial_sample_proportion      : 0.125
 step_size_decrease_rate          : 0.75
 additional_iterations_if_unhealthy : 5
 adagrad_momentum_weighting       : 0.9
 num_tempering_iterations         : 4
 tempering_regularization_start_value : 0.0
 track_exact_loss                 : 0,
 Class                            : RankingFactorizationRecommender
 
 Schema
 ------
 User ID                          : STATE
 Item ID                          : WEATHER
 Target                           : FATALS
 Additional observation features  : 18
 User side features               : []
 Item side features               : []
 
 Statistics
 ----------
 Number of observations           : 23035
 Number of users                  : 50
 Number of items                  : 9
 
 Training summary
 ----------------
 Training time                    : 9.5308
 
 Model Parameters
 ----------------
 Model class                      : RankingFactorizationRecommender
 num_factors                      : 32
 binary_target                    : 0
 side_data_factorization          : 1
 solver                           : auto
 nmf                              : 0
 max_iterations                   : 50
 
 Regularization Settings
 -----------------------
 regularization                   : 0.001
 regularization_type              : normal
 linear_regularization            : 0.001
 ranking_regularization           : 0.1
 unobserved_rating_value          : -1.79769313486e+308
 num_sampled_negative_examples    : 8
 ials_confidence_scaling_type     : auto
 ials_confidence_scaling_factor   : 1
 
 Optimization Settings
 ---------------------
 init_random_sigma                : 0.01
 sgd_convergence_interval         : 4
 sgd_convergence_threshold        : 0.0
 sgd_max_trial_iterations         : 5
 sgd_sampling_block_size          : 131072
 sgd_step_adjustment_interval     : 4
 sgd_step_size                    : 0.0
 sgd_trial_sample_minimum_size    : 10000
 sgd_trial_sample_proportion      : 0.125
 step_size_decrease_rate          : 0.75
 additional_iterations_if_unhealthy : 5
 adagrad_momentum_weighting       : 0.9
 num_tempering_iterations         : 4
 tempering_regularization_start_value : 0.0
 track_exact_loss                 : 0,
 Class                            : RankingFactorizationRecommender
 
 Schema
 ------
 User ID                          : STATE
 Item ID                          : WEATHER
 Target                           : FATALS
 Additional observation features  : 18
 User side features               : []
 Item side features               : []
 
 Statistics
 ----------
 Number of observations           : 23035
 Number of users                  : 50
 Number of items                  : 9
 
 Training summary
 ----------------
 Training time                    : 5.4758
 
 Model Parameters
 ----------------
 Model class                      : RankingFactorizationRecommender
 num_factors                      : 12
 binary_target                    : 0
 side_data_factorization          : 1
 solver                           : auto
 nmf                              : 0
 max_iterations                   : 25
 
 Regularization Settings
 -----------------------
 regularization                   : 0.001
 regularization_type              : normal
 linear_regularization            : 0.001
 ranking_regularization           : 0.1
 unobserved_rating_value          : -1.79769313486e+308
 num_sampled_negative_examples    : 8
 ials_confidence_scaling_type     : auto
 ials_confidence_scaling_factor   : 1
 
 Optimization Settings
 ---------------------
 init_random_sigma                : 0.01
 sgd_convergence_interval         : 4
 sgd_convergence_threshold        : 0.0
 sgd_max_trial_iterations         : 5
 sgd_sampling_block_size          : 131072
 sgd_step_adjustment_interval     : 4
 sgd_step_size                    : 0.0
 sgd_trial_sample_minimum_size    : 10000
 sgd_trial_sample_proportion      : 0.125
 step_size_decrease_rate          : 0.75
 additional_iterations_if_unhealthy : 5
 adagrad_momentum_weighting       : 0.9
 num_tempering_iterations         : 4
 tempering_regularization_start_value : 0.0
 track_exact_loss                 : 0,
 Class                            : RankingFactorizationRecommender
 
 Schema
 ------
 User ID                          : STATE
 Item ID                          : WEATHER
 Target                           : FATALS
 Additional observation features  : 18
 User side features               : []
 Item side features               : []
 
 Statistics
 ----------
 Number of observations           : 23035
 Number of users                  : 50
 Number of items                  : 9
 
 Training summary
 ----------------
 Training time                    : 6.62
 
 Model Parameters
 ----------------
 Model class                      : RankingFactorizationRecommender
 num_factors                      : 32
 binary_target                    : 0
 side_data_factorization          : 1
 solver                           : auto
 nmf                              : 0
 max_iterations                   : 25
 
 Regularization Settings
 -----------------------
 regularization                   : 0.001
 regularization_type              : normal
 linear_regularization            : 0.001
 ranking_regularization           : 0.5
 unobserved_rating_value          : -1.79769313486e+308
 num_sampled_negative_examples    : 4
 ials_confidence_scaling_type     : auto
 ials_confidence_scaling_factor   : 1
 
 Optimization Settings
 ---------------------
 init_random_sigma                : 0.01
 sgd_convergence_interval         : 4
 sgd_convergence_threshold        : 0.0
 sgd_max_trial_iterations         : 5
 sgd_sampling_block_size          : 131072
 sgd_step_adjustment_interval     : 4
 sgd_step_size                    : 0.0
 sgd_trial_sample_minimum_size    : 10000
 sgd_trial_sample_proportion      : 0.125
 step_size_decrease_rate          : 0.75
 additional_iterations_if_unhealthy : 5
 adagrad_momentum_weighting       : 0.9
 num_tempering_iterations         : 4
 tempering_regularization_start_value : 0.0
 track_exact_loss                 : 0]

Based on the training time, the second model seems to be the most efficient. As most of the other parameters do not change much between models.

This project is also available along with the data set used in Github.