TypeError: unsupported operand type(s) in .tools.iamc with numexpr #421
Description
@OFR-IIASA reported the following exception, which occurred via use of .tools.costs.create_cost_projections():
Full traceback:
...create_projections_gdp Adjust ratios using GDP data
genno.caching.cached_load 'cache_path' configuration not set; using C:\Users\fricko\AppData\Local\genno\genno\Cache
...cached_load Cache miss for iamc_like_data_for_query(<09536725…>)
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "C:\Users\fricko\AppData\Local\anaconda3\Scripts\mix-models.exe\__main__.py", line 7, in <module>
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\click\core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\click\decorators.py", line 45, in new_func
return f(get_current_context().obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Repo\message_data_ssp\message_data\projects\ssp_dev\__init__.py", line 159, in run
runscript_main(context)
File "C:\Repo\message_data_ssp\message_data\projects\ssp_dev\runscript_main.py", line 452, in __init__
self.add_tec_cost()
File "C:\Repo\message_data_ssp\message_data\projects\ssp_dev\runscript_main.py", line 1084, in add_tec_cost
inv, fix = gen_te_projections(
^^^^^^^^^^^^^^^^^^^
File "C:\Repo\message-ix-models\message_ix_models\model\material\data_util.py", line 817, in gen_te_projections
out_materials = create_cost_projections(cfg)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Repo\message-ix-models\message_ix_models\tools\costs\projections.py", line 682, in create_cost_projections
df_costs = func(config)
^^^^^^^^^^^^
File "C:\Repo\message-ix-models\message_ix_models\tools\costs\projections.py", line 165, in create_projections_gdp
adjust_cost_ratios_with_gdp(df_region_diff, config)
File "C:\Repo\message-ix-models\message_ix_models\tools\costs\gdp.py", line 172, in adjust_cost_ratios_with_gdp
process_raw_ssp_data(context, config)
File "C:\Repo\message-ix-models\message_ix_models\tools\costs\gdp.py", line 120, in process_raw_ssp_data
result = c.get(k_result)
^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\genno\core\computer.py", line 691, in get
raise ComputationError(exc) from None
genno.core.exceptions.ComputationError: computing '_pop 5:n-y:message_ix_models.project.ssp.data.SSPUpdate' using:
(<bound method SSPDataSource.get of <message_ix_models.project.ssp.data.SSPUpdate object at 0x0000016ED4AA16D0>>,)
Use Computer.describe(...) to trace the computation.
Computation traceback:
File "C:\Repo\message-ix-models\message_ix_models\project\ssp\data.py", line 82, in get
return iamc_like_data_for_query(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\genno\caching.py", line 224, in cached_load
return _write(path, func(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^
File "C:\Repo\message-ix-models\message_ix_models\tools\iamc.py", line 174, in iamc_like_data_for_query
return to_quantity(
^^^^^^^^^^^^
File "C:\Repo\message-ix-models\message_ix_models\tools\iamc.py", line 249, in to_quantity
.query(query)
^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\pandas\core\frame.py", line 4823, in query
res = self.eval(expr, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\pandas\core\frame.py", line 4949, in eval
return _eval(expr, inplace=inplace, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\pandas\core\computation\eval.py", line 357, in eval
ret = eng_inst.evaluate()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\pandas\core\computation\engines.py", line 81, in evaluate
res = self._evaluate()
^^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\pandas\core\computation\engines.py", line 121, in _evaluate
return ne.evaluate(s, local_dict=scope)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\numexpr\necompiler.py", line 975, in evaluate
raise e
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\numexpr\necompiler.py", line 872, in validate
_names_cache[expr_key] = getExprNames(ex, context, sanitize=sanitize)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\numexpr\necompiler.py", line 721, in getExprNames
ex = stringToExpression(text, {}, context, sanitize)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\fricko\AppData\Local\anaconda3\Lib\site-packages\numexpr\necompiler.py", line 307, in stringToExpression
ex = eval(c, names)
^^^^^^^^^^^^^^
File "<expr>", line 1, in <module>
TypeError: unsupported operand type(s) for |: 'bool' and 'OpNode'
Specific conditions where the error occurs and does not occur:
- For @OFR-IIASA:
- Error with pandas 2.2.2 and numexpr 2.8.7
- Error with pandas 2.2.3 and numexpr 2.8.7.
- Error with pandas 2.2.3 and numexpr 2.12.1.
- No error with pandas 2.2.3 and numexpr not installed.
- For @macflo8:
- No error with pandas 2.2.3 and numexpr not installed.
- No error with pandas 2.2.3 and numexpr 2.12.1.
- On our "pytest" workflow/GitHub Actions, here:
- No error with pandas 2.2.3 and numexpr not installed.
Thus:
- A workaround for this particular exception from numexpr seems to be to uninstall the package.
- It is unclear (a) why the code breaks when numexpr is installed, but (b) apparently not in certain cases tested by Florian.