Panel and interpretation of results

Hi all,

nice to try this fantastic algorithm again.

So, to keep it simple, let's say I have the dataset attached:
Prove.xlsx

In the dataset I have 7 products denoted with an id: 6068, 6073, 6097, 6099, 6101, 6103, 6104. They are observed in 12 time periods each, respectively from 2004 to 2015. For each product, in each year I observe three variables: price, stdunits (which is the demand) and recalls. Recalls is a dummy taking 1 if the product has been withdrawn from the market and 0 otherwise. I wish to compute the cross price elasticities of the 7 products. Ideally I would like to end up with a 7x7 matrix (let's call it K) of coefficients where elements K[i,j] represent the cross elasticity of product I with respect to product j and K[i,i] (i.e. elements on the main diagonal) the own-price elasticities.

Hence, first question is: is this possible?

If not, I have improved an already discussed code. As far as I've understood, by forming ARBITRARY groups (groups are needed just to work more easily on code at least as far as I have understood), it's possible to estimate the coefficients of own-price elasticity (for each good) and of the average price of the other (other than the i^th) products (i.e. a matrix of 7x2 coefficients).
The code is implemented as follows (note the following is with 15 rather than 7 products):

from econml.dml import LinearDMLCateEstimator
from sklearn.preprocessing import PolynomialFeatures
from econml.inference import StatsModelsInference
n_products_per_group = 2
n_years = 12
n_groups = 8
n_products = n_groups * n_products_per_group
# p[i, j] = price of product i on year j
p = prices_unstacked.values
# p_minus[i, j] = average price of products other than i within the same group on year j
p_minus = np.zeros(p.shape)
# q[i, j] = demand of product i on year j
q = quantities_unstacked.values
# r[i, j] = recalls of product i on year j
r = recalls_unstacked.values
# X[i, j] = (one-hot-encoding of product i) exludign product 0 used as a baseline
X = np.zeros((n_products, n_years, n_products + n_years -1))
# Adding some time effects on the price of each product
for g in np.arange(n_groups):
    for i in np.arange(n_products_per_group):
        for j in np.arange(n_years):
            index_i = n_products_per_group*g + i
            p[index_i, j] += 1.0*(j==1)

# Creating fake demands
for g in np.arange(n_groups):
    for i in np.arange(n_products_per_group):
        for j in np.arange(n_years):
            minus_i = (np.arange(n_products) >= (n_products_per_group*g))
            minus_i &= (np.arange(n_products) < (n_products_per_group*(g+1))) #&= is the intersection operator
            minus_i &= (np.arange(n_products) != (n_products_per_group*g + i))
            index_i = n_products_per_group*g + i
            p_minus[index_i, j] = np.mean(p[minus_i, j])
            # adding some time fixed effect on the demand
            q[index_i, j] += 1.0 * (j==1)
            # adding individual fixed effects
            X[index_i, j, :(n_products-1)] = 1.0 * (np.arange(1, n_products) == (index_i))
            #recalls
            X[index_i, j, n_products-1:n_products] =r[index_i][j]
            # adding the one-hot encoding of the month as part of the features of the observation
            X[index_i, j, (n_products):] =1.0 * (np.arange(1, n_years) == j)
            # adding the recalls: is the very last value of each row in X[index_i]


# Creating the inputs to the DML method
Y = q.flatten()
T = np.hstack([p.reshape((n_products*n_years, 1)), p_minus.reshape((n_products*n_years, 1))])
X = X.reshape((n_products*n_years, n_products + n_years - 1))

from sklearn.linear_model import MultiTaskLassoCV, LassoCV, LinearRegression
# Set linear_first_stages to false because this creates extra expanded features of the form X cross W 
# that slows code down. Effect on correctness is small. For full correctness it must be set to True
# if linear models are used as first stage models. You can set to False if for instance Random Forests
# are used as first stages.
est = LinearDMLCateEstimator(model_y = LassoCV(cv=3),
                             model_t = MultiTaskLassoCV(cv=3),
                             linear_first_stages=False)
# Using the one-hot encondings of the products as X and the one-hot encodings of the month as W
# to control for time fixed effects
est.fit(Y, T, X[:, :(n_products)], X[:,(n_products-1):], inference='statsmodels')
coeff_tf = est.const_marginal_effect(np.eye(n_products))

lc, uc = est.const_marginal_effect_interval(np.eye(n_products), alpha=.05)
arr_tf = np.stack((coeff_tf, lc,uc), axis=-1)
# Cool way to visualize results: point estimate and the upper and lower confidence bounds for the cross-price elasticity matrix.
print(np.array(["{:.4f} ({:.4f}, {:.4f})".format(arr_tf[i,j,0], arr_tf[i,j,1], arr_tf[i,j,2]) for i in range(arr_tf.shape[0]) for j in range(arr_tf.shape[1])]).reshape(lc.shape))

My questions are, therefore: is the code above correctly implemented? If so, how can I interpret the results? Is there a way to form two coefficient matrices one for recalls = 0 and the other for recalls = 1?

Thank you very much,

Federico

No repository files or tests are named. Begin with the LinearDMLCateEstimator fit, const_marginal_effect, and const_marginal_effect_interval entry points shown in the notebook, then determine whether the requested elasticity matrices and recall-specific effects are supported; done means providing a maintainer-confirmed interpretation or a clearly scoped implementation plan.