Spearman's Rank correlation does take a while on a Mac

I could not make it work with numpy alone and had to use scipy.stats. I needed Pandas to upload my data. I am trying to learn Python and thought some members may be interested in getting Spearman’s rank correlation from P123’s downloads…

It does take a while as Pitmaster says. Are they using a" bubble sort?" :slightly_smiling_face: Actually, I doubt that but it does take a while. 90 second for 44 features on a Mac. 21 years of weekly data from a DataMiner download.

Notice that if you are not using buckets to aggregate the data the rank correlations will be smaller numbers:

Addendum (just the code if you are not interested):

import pandas as pd
import numpy as np
from scipy.stats import spearmanr
import time

(#) Assuming df is your DataFrame after concatenation

(#) Start the timer
start_time = time.time()

(#) Prepare a list to store the results
results_list =

(#) Exclude ‘ExcessReturn’ from the loop
columns_to_consider = df.columns.drop(‘ExcessReturn’)

for col in columns_to_consider:
(#) Drop rows where either column or ‘ExcessReturn’ has NaN
non_na_df = df[[col, ‘ExcessReturn’]].dropna()

( #) Now both columns will have the same length
correlation, _ = spearmanr(non_na_df[col], non_na_df[‘ExcessReturn’])

(#) Append the result as a tuple to the results_list
results_list.append((col, correlation))

(# )Create a DataFrame from the list of results
correlation_results = pd.DataFrame(results_list, columns=[‘Variable’, ‘Spearman_Correlation_with_ExcessReturn’])

(#) Sort the DataFrame by the correlation values
correlation_results.sort_values(by=‘Spearman_Correlation_with_ExcessReturn’, ascending=False, inplace=True)

(#) End the timer
end_time = time.time()

(#) Print each row to display the feature next to its Spearman’s Rank Correlation
for index, row in correlation_results.iterrows():
print(f"{row[‘Variable’]}: {row[‘Spearman_Correlation_with_ExcessReturn’]}")

(#) Display the execution time
print(f"\nExecution time: {end_time - start_time} seconds")

2 Likes