Imagine we have submission produced by pre-update pipeline.
\nHow to convert it into the new one? Apparently it's not obvious, at least for me.
I think this topic is must for smooth transition of participants from pre-update LB to the current setting.
", "messages": [ { "id": 2878797, "postDate": "2024-06-19T06:53:26.737Z", "content": "It's a new test set. You can't convert. You have to predict on the new test and then use the new weights.
", "rawMarkdown": "It's a new test set. You can't convert. You have to predict on the new test and then use the new weights.", "votes": 3 }, { "id": 2878801, "postDate": "2024-06-19T06:54:11.053Z", "content": "Let's say you have run your old pipeline with new test set and you have created your submission as pandas dataframe
\nnow you cand do something like that
\nALL_TARGET_NAMES = ['ptend_t_0', 'ptend_t_1', 'ptend_t_2', 'ptend_t_3', 'ptend_t_4', 'ptend_t_5', 'ptend_t_6', 'ptend_t_7', 'ptend_t_8', 'ptend_t_9', 'ptend_t_10', 'ptend_t_11', 'ptend_t_12', 'ptend_t_13', 'ptend_t_14', 'ptend_t_15', 'ptend_t_16', 'ptend_t_17', 'ptend_t_18', 'ptend_t_19', 'ptend_t_20', 'ptend_t_21', 'ptend_t_22', 'ptend_t_23', 'ptend_t_24', 'ptend_t_25', 'ptend_t_26', 'ptend_t_27', 'ptend_t_28', 'ptend_t_29', 'ptend_t_30', 'ptend_t_31', 'ptend_t_32', 'ptend_t_33', 'ptend_t_34', 'ptend_t_35', 'ptend_t_36', 'ptend_t_37', 'ptend_t_38', 'ptend_t_39', 'ptend_t_40', 'ptend_t_41', 'ptend_t_42', 'ptend_t_43', 'ptend_t_44', 'ptend_t_45', 'ptend_t_46', 'ptend_t_47', 'ptend_t_48', 'ptend_t_49', 'ptend_t_50', 'ptend_t_51', 'ptend_t_52', 'ptend_t_53', 'ptend_t_54', 'ptend_t_55', 'ptend_t_56', 'ptend_t_57', 'ptend_t_58', 'ptend_t_59', 'ptend_q_0', 'ptend_q_1', 'ptend_q_2', 'ptend_q_3', 'ptend_q_4', 'ptend_q_5', 'ptend_q_6', 'ptend_q_7', 'ptend_q_8', 'ptend_q_9', 'ptend_q_10', 'ptend_q_11', 'ptend_q_12', 'ptend_q_13', 'ptend_q_14', 'ptend_q_15', 'ptend_q_16', 'ptend_q_17', 'ptend_q_18', 'ptend_q_19', 'ptend_q_20', 'ptend_q_21', 'ptend_q_22', 'ptend_q_23', 'ptend_q_24', 'ptend_q_25', 'ptend_q_26', 'ptend_q_27', 'ptend_q_28', 'ptend_q_29', 'ptend_q_30', 'ptend_q_31', 'ptend_q_32', 'ptend_q_33', 'ptend_q_34', 'ptend_q_35', 'ptend_q_36', 'ptend_q_37', 'ptend_q_38', 'ptend_q_39', 'ptend_q_40', 'ptend_q_41', 'ptend_q_42', 'ptend_q_43', 'ptend_q_44', 'ptend_q_45', 'ptend_q_46', 'ptend_q_47', 'ptend_q_48', 'ptend_q_49', 'ptend_q_50', 'ptend_q_51', 'ptend_q_52', 'ptend_q_53', 'ptend_q_54', 'ptend_q_55', 'ptend_q_56', 'ptend_q_57', 'ptend_q_58', 'ptend_q_59', 'ptend_q_0', 'ptend_q_1', 'ptend_q_2', 'ptend_q_3', 'ptend_q_4', 'ptend_q_5', 'ptend_q_6', 'ptend_q_7', 'ptend_q_8', 'ptend_q_9', 'ptend_q_10', 'ptend_q_11', 'ptend_q_12', 'ptend_q_13', 'ptend_q_14', 'ptend_q_15', 'ptend_q_16', 'ptend_q_17', 'ptend_q_18', 'ptend_q_19', 'ptend_q_20', 'ptend_q_21', 'ptend_q_22', 'ptend_q_23', 'ptend_q_24', 'ptend_q_25', 'ptend_q_26', 'ptend_q_27', 'ptend_q_28', 'ptend_q_29', 'ptend_q_30', 'ptend_q_31', 'ptend_q_32', 'ptend_q_33', 'ptend_q_34', 'ptend_q_35', 'ptend_q_36', 'ptend_q_37', 'ptend_q_38', 'ptend_q_39', 'ptend_q_40', 'ptend_q_41', 'ptend_q_42', 'ptend_q_43', 'ptend_q_44', 'ptend_q_45', 'ptend_q_46', 'ptend_q_47', 'ptend_q_48', 'ptend_q_49', 'ptend_q_50', 'ptend_q_51', 'ptend_q_52', 'ptend_q_53', 'ptend_q_54', 'ptend_q_55', 'ptend_q_56', 'ptend_q_57', 'ptend_q_58', 'ptend_q_59', 'ptend_q_0', 'ptend_q_1', 'ptend_q_2', 'ptend_q_3', 'ptend_q_4', 'ptend_q_5', 'ptend_q_6', 'ptend_q_7', 'ptend_q_8', 'ptend_q_9', 'ptend_q_10', 'ptend_q_11', 'ptend_q_12', 'ptend_q_13', 'ptend_q_14', 'ptend_q_15', 'ptend_q_16', 'ptend_q_17', 'ptend_q_18', 'ptend_q_19', 'ptend_q_20', 'ptend_q_21', 'ptend_q_22', 'ptend_q_23', 'ptend_q_24', 'ptend_q_25', 'ptend_q_26', 'ptend_q_27', 'ptend_q_28', 'ptend_q_29', 'ptend_q_30', 'ptend_q_31', 'ptend_q_32', 'ptend_q_33', 'ptend_q_34', 'ptend_q_35', 'ptend_q_36', 'ptend_q_37', 'ptend_q_38', 'ptend_q_39', 'ptend_q_40', 'ptend_q_41', 'ptend_q_42', 'ptend_q_43', 'ptend_q_44', 'ptend_q_45', 'ptend_q_46', 'ptend_q_47', 'ptend_q_48', 'ptend_q_49', 'ptend_q_50', 'ptend_q_51', 'ptend_q_52', 'ptend_q_53', 'ptend_q_54', 'ptend_q_55', 'ptend_q_56', 'ptend_q_57', 'ptend_q_58', 'ptend_q_59', 'ptend_u_0', 'ptend_u_1', 'ptend_u_2', 'ptend_u_3', 'ptend_u_4', 'ptend_u_5', 'ptend_u_6', 'ptend_u_7', 'ptend_u_8', 'ptend_u_9', 'ptend_u_10', 'ptend_u_11', 'ptend_u_12', 'ptend_u_13', 'ptend_u_14', 'ptend_u_15', 'ptend_u_16', 'ptend_u_17', 'ptend_u_18', 'ptend_u_19', 'ptend_u_20', 'ptend_u_21', 'ptend_u_22', 'ptend_u_23', 'ptend_u_24', 'ptend_u_25', 'ptend_u_26', 'ptend_u_27', 'ptend_u_28', 'ptend_u_29', 'ptend_u_30', 'ptend_u_31', 'ptend_u_32', 'ptend_u_33', 'ptend_u_34', 'ptend_u_35', 'ptend_u_36', 'ptend_u_37', 'ptend_u_38', 'ptend_u_39', 'ptend_u_40', 'ptend_u_41', 'ptend_u_42', 'ptend_u_43', 'ptend_u_44', 'ptend_u_45', 'ptend_u_46', 'ptend_u_47', 'ptend_u_48', 'ptend_u_49', 'ptend_u_50', 'ptend_u_51', 'ptend_u_52', 'ptend_u_53', 'ptend_u_54', 'ptend_u_55', 'ptend_u_56', 'ptend_u_57', 'ptend_u_58', 'ptend_u_59', 'ptend_v_0', 'ptend_v_1', 'ptend_v_2', 'ptend_v_3', 'ptend_v_4', 'ptend_v_5', 'ptend_v_6', 'ptend_v_7', 'ptend_v_8', 'ptend_v_9', 'ptend_v_10', 'ptend_v_11', 'ptend_v_12', 'ptend_v_13', 'ptend_v_14', 'ptend_v_15', 'ptend_v_16', 'ptend_v_17', 'ptend_v_18', 'ptend_v_19', 'ptend_v_20', 'ptend_v_21', 'ptend_v_22', 'ptend_v_23', 'ptend_v_24', 'ptend_v_25', 'ptend_v_26', 'ptend_v_27', 'ptend_v_28', 'ptend_v_29', 'ptend_v_30', 'ptend_v_31', 'ptend_v_32', 'ptend_v_33', 'ptend_v_34', 'ptend_v_35', 'ptend_v_36', 'ptend_v_37', 'ptend_v_38', 'ptend_v_39', 'ptend_v_40', 'ptend_v_41', 'ptend_v_42', 'ptend_v_43', 'ptend_v_44', 'ptend_v_45', 'ptend_v_46', 'ptend_v_47', 'ptend_v_48', 'ptend_v_49', 'ptend_v_50', 'ptend_v_51', 'ptend_v_52', 'ptend_v_53', 'ptend_v_54', 'ptend_v_55', 'ptend_v_56', 'ptend_v_57', 'ptend_v_58', 'ptend_v_59', 'cam_out_NETSW', 'cam_out_FLWDS', 'cam_out_PRECSC', 'cam_out_PRECC', 'cam_out_SOLS', 'cam_out_SOLL', 'cam_out_SOLSD', 'cam_out_SOLLD']\n\n\n\nOLD_ALL_TARGET_WEIGHTS = [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ]\n\nsample = pl.read_csv(os.path.join(DATA, ), n_rows=)\nnew_weights = sample[ALL_TARGET_NAMES].to_numpy().flatten()\n\nfor w_new, w_old,t in zip(new_weights, OLD_ALL_TARGET_WEIGHTS, ALL_TARGET_NAMES):\n if w_new != :\n if w_old != :\n submission [t] = submission [t] / w_old\n else: \n submission [t] = \n\nfor idx in range(, ):\n f = f\n t = f\n submission[t] = - df_test[f].to_numpy()/ \nlast loop is necessary because the old weights for ptend_q0002_12-14 were zeros
", "rawMarkdown": "Let's say you have run your old pipeline with new test set and you have created your submission as pandas dataframe\n\n\nnow you cand do something like that\n\n```\nALL_TARGET_NAMES = ['ptend_t_0', 'ptend_t_1', 'ptend_t_2', 'ptend_t_3', 'ptend_t_4', 'ptend_t_5', 'ptend_t_6', 'ptend_t_7', 'ptend_t_8', 'ptend_t_9', 'ptend_t_10', 'ptend_t_11', 'ptend_t_12', 'ptend_t_13', 'ptend_t_14', 'ptend_t_15', 'ptend_t_16', 'ptend_t_17', 'ptend_t_18', 'ptend_t_19', 'ptend_t_20', 'ptend_t_21', 'ptend_t_22', 'ptend_t_23', 'ptend_t_24', 'ptend_t_25', 'ptend_t_26', 'ptend_t_27', 'ptend_t_28', 'ptend_t_29', 'ptend_t_30', 'ptend_t_31', 'ptend_t_32', 'ptend_t_33', 'ptend_t_34', 'ptend_t_35', 'ptend_t_36', 'ptend_t_37', 'ptend_t_38', 'ptend_t_39', 'ptend_t_40', 'ptend_t_41', 'ptend_t_42', 'ptend_t_43', 'ptend_t_44', 'ptend_t_45', 'ptend_t_46', 'ptend_t_47', 'ptend_t_48', 'ptend_t_49', 'ptend_t_50', 'ptend_t_51', 'ptend_t_52', 'ptend_t_53', 'ptend_t_54', 'ptend_t_55', 'ptend_t_56', 'ptend_t_57', 'ptend_t_58', 'ptend_t_59', 'ptend_q0001_0', 'ptend_q0001_1', 'ptend_q0001_2', 'ptend_q0001_3', 'ptend_q0001_4', 'ptend_q0001_5', 'ptend_q0001_6', 'ptend_q0001_7', 'ptend_q0001_8', 'ptend_q0001_9', 'ptend_q0001_10', 'ptend_q0001_11', 'ptend_q0001_12', 'ptend_q0001_13', 'ptend_q0001_14', 'ptend_q0001_15', 'ptend_q0001_16', 'ptend_q0001_17', 'ptend_q0001_18', 'ptend_q0001_19', 'ptend_q0001_20', 'ptend_q0001_21', 'ptend_q0001_22', 'ptend_q0001_23', 'ptend_q0001_24', 'ptend_q0001_25', 'ptend_q0001_26', 'ptend_q0001_27', 'ptend_q0001_28', 'ptend_q0001_29', 'ptend_q0001_30', 'ptend_q0001_31', 'ptend_q0001_32', 'ptend_q0001_33', 'ptend_q0001_34', 'ptend_q0001_35', 'ptend_q0001_36', 'ptend_q0001_37', 'ptend_q0001_38', 'ptend_q0001_39', 'ptend_q0001_40', 'ptend_q0001_41', 'ptend_q0001_42', 'ptend_q0001_43', 'ptend_q0001_44', 'ptend_q0001_45', 'ptend_q0001_46', 'ptend_q0001_47', 'ptend_q0001_48', 'ptend_q0001_49', 'ptend_q0001_50', 'ptend_q0001_51', 'ptend_q0001_52', 'ptend_q0001_53', 'ptend_q0001_54', 'ptend_q0001_55', 'ptend_q0001_56', 'ptend_q0001_57', 'ptend_q0001_58', 'ptend_q0001_59', 'ptend_q0002_0', 'ptend_q0002_1', 'ptend_q0002_2', 'ptend_q0002_3', 'ptend_q0002_4', 'ptend_q0002_5', 'ptend_q0002_6', 'ptend_q0002_7', 'ptend_q0002_8', 'ptend_q0002_9', 'ptend_q0002_10', 'ptend_q0002_11', 'ptend_q0002_12', 'ptend_q0002_13', 'ptend_q0002_14', 'ptend_q0002_15', 'ptend_q0002_16', 'ptend_q0002_17', 'ptend_q0002_18', 'ptend_q0002_19', 'ptend_q0002_20', 'ptend_q0002_21', 'ptend_q0002_22', 'ptend_q0002_23', 'ptend_q0002_24', 'ptend_q0002_25', 'ptend_q0002_26', 'ptend_q0002_27', 'ptend_q0002_28', 'ptend_q0002_29', 'ptend_q0002_30', 'ptend_q0002_31', 'ptend_q0002_32', 'ptend_q0002_33', 'ptend_q0002_34', 'ptend_q0002_35', 'ptend_q0002_36', 'ptend_q0002_37', 'ptend_q0002_38', 'ptend_q0002_39', 'ptend_q0002_40', 'ptend_q0002_41', 'ptend_q0002_42', 'ptend_q0002_43', 'ptend_q0002_44', 'ptend_q0002_45', 'ptend_q0002_46', 'ptend_q0002_47', 'ptend_q0002_48', 'ptend_q0002_49', 'ptend_q0002_50', 'ptend_q0002_51', 'ptend_q0002_52', 'ptend_q0002_53', 'ptend_q0002_54', 'ptend_q0002_55', 'ptend_q0002_56', 'ptend_q0002_57', 'ptend_q0002_58', 'ptend_q0002_59', 'ptend_q0003_0', 'ptend_q0003_1', 'ptend_q0003_2', 'ptend_q0003_3', 'ptend_q0003_4', 'ptend_q0003_5', 'ptend_q0003_6', 'ptend_q0003_7', 'ptend_q0003_8', 'ptend_q0003_9', 'ptend_q0003_10', 'ptend_q0003_11', 'ptend_q0003_12', 'ptend_q0003_13', 'ptend_q0003_14', 'ptend_q0003_15', 'ptend_q0003_16', 'ptend_q0003_17', 'ptend_q0003_18', 'ptend_q0003_19', 'ptend_q0003_20', 'ptend_q0003_21', 'ptend_q0003_22', 'ptend_q0003_23', 'ptend_q0003_24', 'ptend_q0003_25', 'ptend_q0003_26', 'ptend_q0003_27', 'ptend_q0003_28', 'ptend_q0003_29', 'ptend_q0003_30', 'ptend_q0003_31', 'ptend_q0003_32', 'ptend_q0003_33', 'ptend_q0003_34', 'ptend_q0003_35', 'ptend_q0003_36', 'ptend_q0003_37', 'ptend_q0003_38', 'ptend_q0003_39', 'ptend_q0003_40', 'ptend_q0003_41', 'ptend_q0003_42', 'ptend_q0003_43', 'ptend_q0003_44', 'ptend_q0003_45', 'ptend_q0003_46', 'ptend_q0003_47', 'ptend_q0003_48', 'ptend_q0003_49', 'ptend_q0003_50', 'ptend_q0003_51', 'ptend_q0003_52', 'ptend_q0003_53', 'ptend_q0003_54', 'ptend_q0003_55', 'ptend_q0003_56', 'ptend_q0003_57', 'ptend_q0003_58', 'ptend_q0003_59', 'ptend_u_0', 'ptend_u_1', 'ptend_u_2', 'ptend_u_3', 'ptend_u_4', 'ptend_u_5', 'ptend_u_6', 'ptend_u_7', 'ptend_u_8', 'ptend_u_9', 'ptend_u_10', 'ptend_u_11', 'ptend_u_12', 'ptend_u_13', 'ptend_u_14', 'ptend_u_15', 'ptend_u_16', 'ptend_u_17', 'ptend_u_18', 'ptend_u_19', 'ptend_u_20', 'ptend_u_21', 'ptend_u_22', 'ptend_u_23', 'ptend_u_24', 'ptend_u_25', 'ptend_u_26', 'ptend_u_27', 'ptend_u_28', 'ptend_u_29', 'ptend_u_30', 'ptend_u_31', 'ptend_u_32', 'ptend_u_33', 'ptend_u_34', 'ptend_u_35', 'ptend_u_36', 'ptend_u_37', 'ptend_u_38', 'ptend_u_39', 'ptend_u_40', 'ptend_u_41', 'ptend_u_42', 'ptend_u_43', 'ptend_u_44', 'ptend_u_45', 'ptend_u_46', 'ptend_u_47', 'ptend_u_48', 'ptend_u_49', 'ptend_u_50', 'ptend_u_51', 'ptend_u_52', 'ptend_u_53', 'ptend_u_54', 'ptend_u_55', 'ptend_u_56', 'ptend_u_57', 'ptend_u_58', 'ptend_u_59', 'ptend_v_0', 'ptend_v_1', 'ptend_v_2', 'ptend_v_3', 'ptend_v_4', 'ptend_v_5', 'ptend_v_6', 'ptend_v_7', 'ptend_v_8', 'ptend_v_9', 'ptend_v_10', 'ptend_v_11', 'ptend_v_12', 'ptend_v_13', 'ptend_v_14', 'ptend_v_15', 'ptend_v_16', 'ptend_v_17', 'ptend_v_18', 'ptend_v_19', 'ptend_v_20', 'ptend_v_21', 'ptend_v_22', 'ptend_v_23', 'ptend_v_24', 'ptend_v_25', 'ptend_v_26', 'ptend_v_27', 'ptend_v_28', 'ptend_v_29', 'ptend_v_30', 'ptend_v_31', 'ptend_v_32', 'ptend_v_33', 'ptend_v_34', 'ptend_v_35', 'ptend_v_36', 'ptend_v_37', 'ptend_v_38', 'ptend_v_39', 'ptend_v_40', 'ptend_v_41', 'ptend_v_42', 'ptend_v_43', 'ptend_v_44', 'ptend_v_45', 'ptend_v_46', 'ptend_v_47', 'ptend_v_48', 'ptend_v_49', 'ptend_v_50', 'ptend_v_51', 'ptend_v_52', 'ptend_v_53', 'ptend_v_54', 'ptend_v_55', 'ptend_v_56', 'ptend_v_57', 'ptend_v_58', 'ptend_v_59', 'cam_out_NETSW', 'cam_out_FLWDS', 'cam_out_PRECSC', 'cam_out_PRECC', 'cam_out_SOLS', 'cam_out_SOLL', 'cam_out_SOLSD', 'cam_out_SOLLD']\n\n\n# TARGET WEIGHTS\nOLD_ALL_TARGET_WEIGHTS = [30981.265271661872, 22502.432413914863, 18894.14713004499, 14514.244730542465, 10944.348069459196, 9065.01072024503, 9663.669038687454, 12688.557362943708, 19890.17226527665, 25831.37317235381, 33890.367561807274, 44122.94111025334, 59811.25595068309, 79434.07500078829, 107358.80916894016, 135720.8418348218, 149399.8411114814, 128492.95185325432, 91746.23687305572, 72748.76911097553, 66531.53596840335, 62932.30598423903, 56610.26874314136, 49473.14369220607, 43029.18495420936, 36912.67491908133, 31486.93117928144, 26898.072997215502, 23316.638282978325, 20459.73133196152, 18385.68309639014, 17111.405107656312, 16337.80991958771, 15857.759882318944, 15580.902485189716, 15497.59045982052, 15612.2556996736, 15797.88455410361, 15974.218740897895, 16130.395527176632, 16261.310866446129, 16371.892401608216, 16397.019695140876, 16325.463899570548, 16228.641108112768, 16191.809643436269, 16341.207925934068, 16645.711351490587, 17005.493716683693, 17430.29874509864, 17907.24023203076, 18431.55334008694, 19032.471309392287, 19701.355113141435, 20408.236605392685, 20967.20795006453, 21194.427318009974, 21088.521528526755, 19437.91555757985, 13677.902713248171, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 871528441401.8333, 1083221770553.0684, 147034752676.7702, 35556045575.13566, 35153369257.41337, 46086368691.51654, 24689305171.692936, 11343276593.440475, 5396624651.94418, 2449353007.641508, 1132225885.703891, 579547849.1340877, 330219246.7861086, 207613930.3131764, 144580292.27473342, 109933282.92266414, 88706603.092171, 73819777.54163922, 63615988.74519494, 57250262.292053565, 52976073.06761927, 49653169.17819005, 46544975.11484598, 43167606.9599748, 39724375.20499403, 36317177.25886468, 33057511.80930482, 29869089.497658804, 26982386.85583376, 24416235.17215712, 22273651.697369896, 20553426.04804544, 19216240.03357431, 18167694.44812838, 17501855.536957663, 17169938.630597908, 17005382.258644175, 16998475.26752617, 17082890.987979066, 17227982.77516062, 17445823.21630204, 17757404.421785507, 18346092.75160569, 19400573.66632694, 20506722.48296608, 22469648.380506545, 23432031.455169585, 26204163.40545158, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1000000000000000.0, 1000000000000000.0, 1000000000000000.0, 1000000000000000.0, 1000000000000000.0, 1000000000000000.0, 1000000000000000.0, 1000000000000000.0, 1000000000000000.0, 1000000000000000.0, 1000000000000000.0, 3673829810926.31, 371405570725.2526, 14219163611.984406, 3001863018.1934915, 1432766589.9326108, 884599805.0283787, 560127980.1033351, 386052567.7087711, 287331851.051439, 222703657.59538063, 181069239.6264349, 154620864.3164144, 138093777.60284117, 126605828.89875436, 117967840.02553518, 111005814.39518328, 105186901.20678852, 100168133.0295481, 95568646.67416307, 91457433.39515457, 88871610.45308323, 88829796.26374224, 91398113.73291488, 96585131.67000748, 104507692.01463065, 115895119.998433, 131939701.08213414, 154492946.00677127, 183147918.17086875, 215151374.22324687, 247158314.6345976, 266792879.42215955, 279115128.29108113, 370541510.87006927, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 877670509694.7871, 1174826943136.8308, 1270605570069.038, 21727315470.5208, 3159456646.5437946, 1090653401.282219, 727967089.8459107, 384399548.9506704, 290787296.9451616, 232703218.45048887, 197467462.7577736, 174310890.8025987, 160536437.73297343, 153567098.77048483, 152120124.9453068, 153115566.6756177, 153955545.42558223, 153734675.21565756, 154798666.36905554, 163346213.58113608, 180013139.3707387, 200324358.8534948, 220754613.1646765, 241290935.478592, 262868932.2066308, 284448910.01847774, 305681084.4142859, 327605088.8575117, 350473296.7263526, 373964594.1196182, 398396925.8173239, 423528355.65716046, 450447055.544388, 478857006.4973163, 508200335.7126168, 537309657.5789208, 566854568.2904652, 594618842.9455439, 619715928.2391286, 641395460.8414665, 663290039.7810476, 689274894.631561, 718208866.3397261, 743951200.8024124, 761776104.2945968, 772911224.3082078, 804001144.8046833, 772448774.7758856, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4613823.568205323, 1999308.9343799097, 904636.2296014762, 433823.6123842511, 207201.39055371704, 107836.09164720173, 57647.915219220784, 40606.52305039815, 47739.86647922776, 51669.35493930698, 56438.19768395407, 60447.45665200092, 65251.4153955275, 71920.88588011517, 78529.58115204438, 83422.30217897324, 87036.98552475807, 90389.72631774022, 93982.39165674087, 97578.0099352472, 101428.21366062944, 104630.69200130588, 105685.04322626138, 103962.58423268417, 99650.31670632094, 94290.49986206587, 89514.90144353417, 85905.45713126978, 82784.9857650212, 79152.28707014346, 74847.81017353121, 70378.81859610273, 65420.04643792357, 59953.75184604176, 54764.28281143022, 50362.51288353384, 46212.571031725325, 41997.52779088816, 37692.05148110484, 33834.73460995647, 31846.09764364542, 31934.145655397457, 31454.81247448105, 30105.4073072481, 26957.830283611693, 27760.04479210889, 29853.374336459365, 19133.428743715107, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7619940.584531054, 3148394.472742347, 1308415.0022178134, 540515.7720745018, 215237.1053603881, 102546.7276372816, 68453.67122640925, 50692.59053608593, 51487.52043139844, 52104.76838400132, 54019.39151917722, 55856.02168787862, 60347.30240270209, 68990.96019017675, 79096.88768563846, 87574.33453690328, 94158.56052476274, 101903.63670531697, 111746.9753834774, 122460.65399236557, 132086.69387474353, 141041.48571028374, 146354.09441287292, 145953.09590059065, 139496.8007888401, 128508.85108217449, 116665.51769667884, 107458.39706309135, 100259.97236694951, 94108.98505029618, 88439.89456238014, 82734.9027659809, 77061.08621371102, 71333.5319243128, 65999.72532130677, 61798.9972058361, 58237.356419617165, 54715.10266341248, 50825.84431702935, 46059.17688689915, 40740.26050401376, 36335.80228304863, 33981.57568605091, 33589.7143390849, 33988.88524112733, 36272.9364507092, 41183.34413717943, 29194.12369278645, 0.0040536134869726, 0.0138824238058072, 135129884.5084534, 12219717.5342461, 0.0090705273332672, 0.0085898851680217, 0.0215368188774867, 0.0336321308942602]\n\nsample = pl.read_csv(os.path.join(DATA, \"sample_submission.csv\"), n_rows=1)\nnew_weights = sample[ALL_TARGET_NAMES].to_numpy().flatten()\n\nfor w_new, w_old,t in zip(new_weights, OLD_ALL_TARGET_WEIGHTS, ALL_TARGET_NAMES):\n if w_new != 0:\n if w_old != 0:\n submission [t] = submission [t] / w_old\n else: \n submission [t] = 0.0\n \nfor idx in range(12, 15):\n f = f\"state_q0002_{idx}\"\n t = f\"ptend_q0002_{idx}\"\n submission[t] = - df_test[f].to_numpy()/ 1200\n```\n\nlast loop is necessary because the old weights for ptend_q0002_12-14 were zeros\n", "votes": 4 }, { "id": 2878805, "postDate": "2024-06-19T06:58:14.810Z", "content": "Thank you everyone, I was able to solve the problem.
\nThe bug I had most likely had to deal with precision, I think there is a difference between
\n(right version): df[col] / old_weight[col] and
\n(wrong version): 1 / old_weight[col] * df[col] even when using float64
in wrong version you forgot some \"(\": (1 / old_weight[col]) * df[col] 😀
just tested polars version
\n1 / pl.Series(name='weight', values=[5.0, 5.0]) * pl.Series(name='weight', values=[20.0, 25.0]), works like you expect in math (no need for brackets) :D
Imagine we have submission produced by pre-update pipeline.
\nHow to convert it into the new one? Apparently it's not obvious, at least for me.
I think this topic is must for smooth transition of participants from pre-update LB to the current setting.
", "rawMarkdown": "Imagine we have submission produced by pre-update pipeline. \nHow to convert it into the new one? Apparently it's not obvious, at least for me.\n\nI think this topic is must for smooth transition of participants from pre-update LB to the current setting.", "votes": 2 } ], "comments": [ { "id": 2878797, "author_name": "greySnow", "author_url": "", "post_date": "2024-06-19T06:53:26.737000", "content": "It's a new test set. You can't convert. You have to predict on the new test and then use the new weights.
", "votes": 3, "replies": [] }, { "id": 2878801, "author_name": "steubk", "author_url": "", "post_date": "2024-06-19T06:54:11.053000", "content": "Let's say you have run your old pipeline with new test set and you have created your submission as pandas dataframe
\nnow you cand do something like that
\nALL_TARGET_NAMES = ['ptend_t_0', 'ptend_t_1', 'ptend_t_2', 'ptend_t_3', 'ptend_t_4', 'ptend_t_5', 'ptend_t_6', 'ptend_t_7', 'ptend_t_8', 'ptend_t_9', 'ptend_t_10', 'ptend_t_11', 'ptend_t_12', 'ptend_t_13', 'ptend_t_14', 'ptend_t_15', 'ptend_t_16', 'ptend_t_17', 'ptend_t_18', 'ptend_t_19', 'ptend_t_20', 'ptend_t_21', 'ptend_t_22', 'ptend_t_23', 'ptend_t_24', 'ptend_t_25', 'ptend_t_26', 'ptend_t_27', 'ptend_t_28', 'ptend_t_29', 'ptend_t_30', 'ptend_t_31', 'ptend_t_32', 'ptend_t_33', 'ptend_t_34', 'ptend_t_35', 'ptend_t_36', 'ptend_t_37', 'ptend_t_38', 'ptend_t_39', 'ptend_t_40', 'ptend_t_41', 'ptend_t_42', 'ptend_t_43', 'ptend_t_44', 'ptend_t_45', 'ptend_t_46', 'ptend_t_47', 'ptend_t_48', 'ptend_t_49', 'ptend_t_50', 'ptend_t_51', 'ptend_t_52', 'ptend_t_53', 'ptend_t_54', 'ptend_t_55', 'ptend_t_56', 'ptend_t_57', 'ptend_t_58', 'ptend_t_59', 'ptend_q_0', 'ptend_q_1', 'ptend_q_2', 'ptend_q_3', 'ptend_q_4', 'ptend_q_5', 'ptend_q_6', 'ptend_q_7', 'ptend_q_8', 'ptend_q_9', 'ptend_q_10', 'ptend_q_11', 'ptend_q_12', 'ptend_q_13', 'ptend_q_14', 'ptend_q_15', 'ptend_q_16', 'ptend_q_17', 'ptend_q_18', 'ptend_q_19', 'ptend_q_20', 'ptend_q_21', 'ptend_q_22', 'ptend_q_23', 'ptend_q_24', 'ptend_q_25', 'ptend_q_26', 'ptend_q_27', 'ptend_q_28', 'ptend_q_29', 'ptend_q_30', 'ptend_q_31', 'ptend_q_32', 'ptend_q_33', 'ptend_q_34', 'ptend_q_35', 'ptend_q_36', 'ptend_q_37', 'ptend_q_38', 'ptend_q_39', 'ptend_q_40', 'ptend_q_41', 'ptend_q_42', 'ptend_q_43', 'ptend_q_44', 'ptend_q_45', 'ptend_q_46', 'ptend_q_47', 'ptend_q_48', 'ptend_q_49', 'ptend_q_50', 'ptend_q_51', 'ptend_q_52', 'ptend_q_53', 'ptend_q_54', 'ptend_q_55', 'ptend_q_56', 'ptend_q_57', 'ptend_q_58', 'ptend_q_59', 'ptend_q_0', 'ptend_q_1', 'ptend_q_2', 'ptend_q_3', 'ptend_q_4', 'ptend_q_5', 'ptend_q_6', 'ptend_q_7', 'ptend_q_8', 'ptend_q_9', 'ptend_q_10', 'ptend_q_11', 'ptend_q_12', 'ptend_q_13', 'ptend_q_14', 'ptend_q_15', 'ptend_q_16', 'ptend_q_17', 'ptend_q_18', 'ptend_q_19', 'ptend_q_20', 'ptend_q_21', 'ptend_q_22', 'ptend_q_23', 'ptend_q_24', 'ptend_q_25', 'ptend_q_26', 'ptend_q_27', 'ptend_q_28', 'ptend_q_29', 'ptend_q_30', 'ptend_q_31', 'ptend_q_32', 'ptend_q_33', 'ptend_q_34', 'ptend_q_35', 'ptend_q_36', 'ptend_q_37', 'ptend_q_38', 'ptend_q_39', 'ptend_q_40', 'ptend_q_41', 'ptend_q_42', 'ptend_q_43', 'ptend_q_44', 'ptend_q_45', 'ptend_q_46', 'ptend_q_47', 'ptend_q_48', 'ptend_q_49', 'ptend_q_50', 'ptend_q_51', 'ptend_q_52', 'ptend_q_53', 'ptend_q_54', 'ptend_q_55', 'ptend_q_56', 'ptend_q_57', 'ptend_q_58', 'ptend_q_59', 'ptend_q_0', 'ptend_q_1', 'ptend_q_2', 'ptend_q_3', 'ptend_q_4', 'ptend_q_5', 'ptend_q_6', 'ptend_q_7', 'ptend_q_8', 'ptend_q_9', 'ptend_q_10', 'ptend_q_11', 'ptend_q_12', 'ptend_q_13', 'ptend_q_14', 'ptend_q_15', 'ptend_q_16', 'ptend_q_17', 'ptend_q_18', 'ptend_q_19', 'ptend_q_20', 'ptend_q_21', 'ptend_q_22', 'ptend_q_23', 'ptend_q_24', 'ptend_q_25', 'ptend_q_26', 'ptend_q_27', 'ptend_q_28', 'ptend_q_29', 'ptend_q_30', 'ptend_q_31', 'ptend_q_32', 'ptend_q_33', 'ptend_q_34', 'ptend_q_35', 'ptend_q_36', 'ptend_q_37', 'ptend_q_38', 'ptend_q_39', 'ptend_q_40', 'ptend_q_41', 'ptend_q_42', 'ptend_q_43', 'ptend_q_44', 'ptend_q_45', 'ptend_q_46', 'ptend_q_47', 'ptend_q_48', 'ptend_q_49', 'ptend_q_50', 'ptend_q_51', 'ptend_q_52', 'ptend_q_53', 'ptend_q_54', 'ptend_q_55', 'ptend_q_56', 'ptend_q_57', 'ptend_q_58', 'ptend_q_59', 'ptend_u_0', 'ptend_u_1', 'ptend_u_2', 'ptend_u_3', 'ptend_u_4', 'ptend_u_5', 'ptend_u_6', 'ptend_u_7', 'ptend_u_8', 'ptend_u_9', 'ptend_u_10', 'ptend_u_11', 'ptend_u_12', 'ptend_u_13', 'ptend_u_14', 'ptend_u_15', 'ptend_u_16', 'ptend_u_17', 'ptend_u_18', 'ptend_u_19', 'ptend_u_20', 'ptend_u_21', 'ptend_u_22', 'ptend_u_23', 'ptend_u_24', 'ptend_u_25', 'ptend_u_26', 'ptend_u_27', 'ptend_u_28', 'ptend_u_29', 'ptend_u_30', 'ptend_u_31', 'ptend_u_32', 'ptend_u_33', 'ptend_u_34', 'ptend_u_35', 'ptend_u_36', 'ptend_u_37', 'ptend_u_38', 'ptend_u_39', 'ptend_u_40', 'ptend_u_41', 'ptend_u_42', 'ptend_u_43', 'ptend_u_44', 'ptend_u_45', 'ptend_u_46', 'ptend_u_47', 'ptend_u_48', 'ptend_u_49', 'ptend_u_50', 'ptend_u_51', 'ptend_u_52', 'ptend_u_53', 'ptend_u_54', 'ptend_u_55', 'ptend_u_56', 'ptend_u_57', 'ptend_u_58', 'ptend_u_59', 'ptend_v_0', 'ptend_v_1', 'ptend_v_2', 'ptend_v_3', 'ptend_v_4', 'ptend_v_5', 'ptend_v_6', 'ptend_v_7', 'ptend_v_8', 'ptend_v_9', 'ptend_v_10', 'ptend_v_11', 'ptend_v_12', 'ptend_v_13', 'ptend_v_14', 'ptend_v_15', 'ptend_v_16', 'ptend_v_17', 'ptend_v_18', 'ptend_v_19', 'ptend_v_20', 'ptend_v_21', 'ptend_v_22', 'ptend_v_23', 'ptend_v_24', 'ptend_v_25', 'ptend_v_26', 'ptend_v_27', 'ptend_v_28', 'ptend_v_29', 'ptend_v_30', 'ptend_v_31', 'ptend_v_32', 'ptend_v_33', 'ptend_v_34', 'ptend_v_35', 'ptend_v_36', 'ptend_v_37', 'ptend_v_38', 'ptend_v_39', 'ptend_v_40', 'ptend_v_41', 'ptend_v_42', 'ptend_v_43', 'ptend_v_44', 'ptend_v_45', 'ptend_v_46', 'ptend_v_47', 'ptend_v_48', 'ptend_v_49', 'ptend_v_50', 'ptend_v_51', 'ptend_v_52', 'ptend_v_53', 'ptend_v_54', 'ptend_v_55', 'ptend_v_56', 'ptend_v_57', 'ptend_v_58', 'ptend_v_59', 'cam_out_NETSW', 'cam_out_FLWDS', 'cam_out_PRECSC', 'cam_out_PRECC', 'cam_out_SOLS', 'cam_out_SOLL', 'cam_out_SOLSD', 'cam_out_SOLLD']\n\n\n\nOLD_ALL_TARGET_WEIGHTS = [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ]\n\nsample = pl.read_csv(os.path.join(DATA, ), n_rows=)\nnew_weights = sample[ALL_TARGET_NAMES].to_numpy().flatten()\n\nfor w_new, w_old,t in zip(new_weights, OLD_ALL_TARGET_WEIGHTS, ALL_TARGET_NAMES):\n if w_new != :\n if w_old != :\n submission [t] = submission [t] / w_old\n else: \n submission [t] = \n\nfor idx in range(, ):\n f = f\n t = f\n submission[t] = - df_test[f].to_numpy()/ \nlast loop is necessary because the old weights for ptend_q0002_12-14 were zeros
", "votes": 4, "replies": [] }, { "id": 2878805, "author_name": "slime", "author_url": "", "post_date": "2024-06-19T06:58:14.810000", "content": "Thank you everyone, I was able to solve the problem.
\nThe bug I had most likely had to deal with precision, I think there is a difference between
\n(right version): df[col] / old_weight[col] and
\n(wrong version): 1 / old_weight[col] * df[col] even when using float64
in wrong version you forgot some \"(\": (1 / old_weight[col]) * df[col] 😀
just tested polars version
\n1 / pl.Series(name='weight', values=[5.0, 5.0]) * pl.Series(name='weight', values=[20.0, 25.0]), works like you expect in math (no need for brackets) :D