Repository navigation
[BUG] Potentially malicious code on commandline detection with PSC v4.3.0 #3928
Description
Describe the bug
Up until today, we were running Enterprise Security 8.3.0 in Splunk Cloud, with Splunk_ML_Toolkit 5.7.0 and Splunk_SA_Scientific_Python_linux_x86_64 from 4.3.0.
We recently noticed failed jobs for the Potentially malicious code on commandline detection.
The apply query fails with this error:
We immediatelly downgraded to Splunk_ML_Toolkit 5.6.4 and Splunk_SA_Scientific_Python_linux_x86_64 from 4.2.3, and this solved the issue.
As you may already know, starting with ES 8.2.3+, this comes comes with Splunk_ML_Toolkit version 5.6.0 preinstalled.
I believe your suggestion is to use [version 5.5.0],(https://github.com/splunk/security_content/blob/v5.22.0/contentctl.yml#L176) and version 4.2.2, respectively.
But this conflicts with ES 8.2.3+ requirements.
search.log:
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last):
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/processors/ApplyProcessor.py", line 169, in apply
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: prediction_df = algo.apply(df, process_options)
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/base.py", line 331, in apply
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: y_hat = self.estimator.predict(X.values)
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/4_3_0/lib/python3.13/site-packages/sklearn/linear_model/_base.py", line 297, in predict
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: return self._decision_function(X)
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: ~~~~~~~~~~~~~~~~~~~~~~~^^^
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/4_3_0/lib/python3.13/site-packages/sklearn/linear_model/_base.py", line 274, in _decision_function
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: check_is_fitted(self)
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: ~~~~~~~~~~~~~~~^^^^^^
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/4_3_0/lib/python3.13/site-packages/sklearn/utils/validation.py", line 1751, in check_is_fitted
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: tags = get_tags(estimator)
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/4_3_0/lib/python3.13/site-packages/sklearn/utils/_tags.py", line 398, in get_tags
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: tags = estimator.__sklearn_tags__()
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/4_3_0/lib/python3.13/site-packages/sklearn/linear_model/_base.py", line 692, in __sklearn_tags__
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: tags.input_tags.sparse = not self.positive
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: ^^^^^^^^^^^^^
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: AttributeError: 'LinearRegression' object has no attribute 'positive'
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: WARNING Error while applying model "unusual_commandline_detection": 'LinearRegression' object has no attribute 'positive'
03-02-2026 10:55:46.218 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: 'LinearRegression' object has no attribute 'positive'
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last):
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/processors/ApplyProcessor.py", line 169, in apply
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: prediction_df = algo.apply(df, process_options)
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/base.py", line 331, in apply
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: y_hat = self.estimator.predict(X.values)
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/4_3_0/lib/python3.13/site-packages/sklearn/linear_model/_base.py", line 297, in predict
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: return self._decision_function(X)
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: ~~~~~~~~~~~~~~~~~~~~~~~^^^
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/4_3_0/lib/python3.13/site-packages/sklearn/linear_model/_base.py", line 274, in _decision_function
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: check_is_fitted(self)
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: ~~~~~~~~~~~~~~~^^^^^^
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/4_3_0/lib/python3.13/site-packages/sklearn/utils/validation.py", line 1751, in check_is_fitted
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: tags = get_tags(estimator)
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/4_3_0/lib/python3.13/site-packages/sklearn/utils/_tags.py", line 398, in get_tags
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: tags = estimator.__sklearn_tags__()
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64/4_3_0/lib/python3.13/site-packages/sklearn/linear_model/_base.py", line 692, in __sklearn_tags__
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: tags.input_tags.sparse = not self.positive
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: ^^^^^^^^^^^^^
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: AttributeError: 'LinearRegression' object has no attribute 'positive'
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: During handling of the above exception, another exception occurred:
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: Traceback (most recent call last):
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/cexc/__init__.py", line 174, in run
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: while self._handle_chunk():
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: ~~~~~~~~~~~~~~~~~~^^
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/cexc/__init__.py", line 236, in _handle_chunk
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: ret = self.handler(metadata, body)
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/apply.py", line 519, in handler
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: self.controller.execute()
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: ~~~~~~~~~~~~~~~~~~~~~~~^^
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/chunked_controller.py", line 292, in execute
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: self.processor.process()
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: ~~~~~~~~~~~~~~~~~~~~~~^^
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/processors/ApplyProcessor.py", line 192, in process
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: self.df = self.apply(self.df, self.algo, self.process_options)
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: File "/opt/splunk/etc/apps/Splunk_ML_Toolkit/bin/processors/ApplyProcessor.py", line 177, in apply
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: raise RuntimeError(e)
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962199 ChunkedExternProcessorStderrLogger] - stderr: RuntimeError: 'LinearRegression' object has no attribute 'positive'
03-02-2026 10:55:46.220 ERROR ChunkedExternProcessor [962236 phase_2] - Error in 'apply' command: 'LinearRegression' object has no attribute 'positive'
03-02-2026 10:55:46.241 INFO ReducePhaseExecutor [962236 phase_2] - Not downloading remote search.log files. Reason: Remote log download is currently disabled.
03-02-2026 10:55:46.241 INFO ReducePhaseExecutor [962236 phase_2] - Not downloading remote search_telemetry.json files. Reason: fetch_remote_search_telemetry=false.
03-02-2026 10:55:46.241 INFO ReducePhaseExecutor [962236 phase_2] - Ending phase_2
03-02-2026 10:55:46.241 INFO ReducePhaseExecutor [962222 StatusEnforcerThread] - ReducePhaseExecutor=1 action=QUIT
03-02-2026 10:55:46.241 INFO DispatchExecutor [962222 StatusEnforcerThread] - Search applied action=QUIT while status=GROUND
03-02-2026 10:55:46.241 INFO SearchStatusEnforcer [962222 StatusEnforcerThread] - sid=1772448913.3973_1FD5CB9C-2F59-41C7-8872-AA45FE2439F0, newState=FAILED, message=Error in 'apply' command: 'LinearRegression' object has no attribute 'positive'
03-02-2026 10:55:46.241 ERROR SearchStatusEnforcer [962222 StatusEnforcerThread] - SearchMessage orig_component=SearchStatusEnforcer sid=1772448913.3973_1FD5CB9C-2F59-41C7-8872-AA45FE2439F0 message_key= message=Error in 'apply' command: 'LinearRegression' object has no attribute 'positive'
03-02-2026 10:55:46.241 INFO SearchStatusEnforcer [962222 StatusEnforcerThread] - State changed to FAILED: Error in 'apply' command: 'LinearRegression' object has no attribute 'positive'
Expected behavior
A clear and concise description of what you expected to happen.
App Version:
- ESCU: v5.22.0
- Splunk Security Essentials: v3.8.3
Additional context
Add any other context about the problem here.
Activity
patel-bhavin commentedon Mar 6, 2026
Hello @AndreiBanaru - We’ve encountered this issue as well, and we’re actively discussing it internally as part of ongoing detection maintenance. At the moment we don’t have a confirmed near-term fix—this appears to be tied to library/runtime compatibility. We’re also evaluating whether these ML-based detections are better handled in Splunk UEBA (given stronger ML capabilities and lifecycle support) and potentially deprecating them in ESCU due to limited maintenance and testing coverage.
Also curious: how extensively are you leveraging the ESCU ML searches in your environment?
AndreiBanaru commentedon Mar 6, 2026
Hey @patel-bhavin , thanks for acknowledging this.
From the ESCU app, we only have this one active- Potentially malicious code on commandline
nasbench commentedon Mar 16, 2026
MLTK detections will be deprecated in v5.24.0 - See #3922
Reasoning is -
Detection is deprecated as these do not work with the latest Splunk AI Toolkit(5.7.0) and Python for Scientific Computing for Linux 64-bit(4.3.0).Hence, I will be closing this issue as complete.